URL categorization of websites\u200a\u2014\u200acybersecurity<\/strong><\/p>\n An important objective for URL categorization arises in the context of cybersecurity, where we want to protect our systems and sensitive information from digital attacks.<\/p>\n A typical example of latter is the practice of phishing\u200a\u2014\u200acreating a counterfeit website that tries to mimic the genuine website with the purpose of obtaining critical information from the user.<\/p>\n Phishing attacks can use a variety of methods, from link manipulation (making malicious URL look like legitimate URL), evasion of filters (using images of websites), forgery of website (injecting javascript or other code at legitimate website), social engineering and others.<\/p>\n When building a machine learning mode for malicious websites identification, a given URL may be binary classified as either \u201cmalicious\/not safe\u201d or \u201csafe\u201d.<\/p>\n There are many other ways to classify URL, one of them, content categorization, will be discussed in more detail in the second post.<\/p>\n This is part 1 of our multi-post article that will focus on several different topics:<\/p>\n The URL classification is usually done on a large number of websites, thus it is almost always automated. With the rise of machine learning and deep learning models in last decade, the automation for URL classification is thus usually done by a machine learning model.<\/p>\n Training data set for URL categorization ML models When building a supervised machine learning model, the first step is obtaining the necessary labelled data\u200a\u2014\u200atraining data set, on which to build the ML model.<\/p>\n In our case, what we need is a list of malicious URLs that have been manually checked and labelled as such.<\/p>\n One great source for this purpose can be found here: https:\/\/www.unb.ca\/cic\/datasets\/url-2016.html<\/a>.<\/p>\n We need two classes of URLS (\u201csafe\u201d and \u201cnot safe), the authors collected 35,300 URLs from Alexa top sites that can be considered as safe.<\/p>\n For malicious URLs they selected the following types:<\/p>\n In addition to this, we also collected malicious URLs by using the free API from https:\/\/www.phishtank.com<\/a>.<\/p>\n Feature engineering<\/strong><\/p>\n Once we have the appropriate training data set, the next step is to think about the feature engineering, i.e. defining features that will be computed from our data items which will then be used in the machine learning model as the input.<\/p>\n We will be using features collected from a large number of research papers on this topic (for full list please refer to Appendix\u200a\u2014\u200aResources at the end of the post).<\/p>\n URL features considered were:<\/p>\n Machine Learning Models for URL classification<\/strong><\/p>\n Next step is to consider which machine learning models are appropriate for URL classification model based on these features.<\/p>\n Given the form of features, one can consider the following ML models (though the list is by no means exhaustive):<\/p>\n – random forests<\/p>\n – decision trees<\/p>\n – logistic regression<\/p>\n – adaboost<\/p>\n – xgboost (https:\/\/xgboost.readthedocs.io\/en\/stable\/<\/a>)<\/p>\n – neural nets<\/p>\n During training of ML models it is useful to also use explainability libraries like LIME; Partial Dependence Plot and SHAP (https:\/\/github.com\/slundberg\/shap<\/a>) to better understand which features are most important for driving predictions of the ML model. In our next post, we will do this in practice, when we will build an URL classification model from scratch.<\/p>\n In example above, we considered a specific URL classification model which helps us answer whether a given URL is malicious or not. This service can either be real-time in the sense that it fetches the URL on the fly, as soon as you submit it, then sends it to ML model, which computes the features that we discussed above, then based on these predicts whether we are dealing with problematic URL or not.<\/p>\n It can however also work from the offline data set of malicious URLs, where the URL submitted could be just checked against existing offline URL database. The advantage of this is that it is much faster, because you do not have to wait for the URL to be fetched and processed. So many applications actually use offline URLs database.<\/p>\n Advantage of real-time URL classification however is that you can classify URLs that were not seen before. Or e.g. published just a short while ago, so the URL classification bots of the service have not yet checked it. This approach is also a bit more safe because whereas some URL could be safe e.g. 1 week ago, it could be hijacked in the meantime and is not safe anymore.<\/p>\n URL categorization is not used only for analysing safety of websites. One very common use is to do a URL category check in terms of URL content.<\/p>\n A company e.g. may be interested in not allowing employees use shopping or gaming websites so a filtering system set up needs to know the content category of each domain that may be visited. There are two options available here, one, most often is to use an offline content categorization database of domains (there are more than 350+ million domains on the web) and the domain of URL requested by the user is then quickly checked for category against this this database.<\/p>\n So if user wants to visits Netflix and the content category that comes back from DB is TV, then the filtering system would block access.<\/p>\n Offline content categorization databases can be used for other purposes as well. E.g. a platform with millions of domains would like to provide information about the content category of each domain to the users of its platform.<\/p>\n Appendix<\/strong><\/p>\n Resources:<\/p>\n https:\/\/arxiv.org\/pdf\/2009.11116.pdf<\/a><\/p>\n https:\/\/www.igi-global.com\/article\/phishing-website-detection-with-semantic-features-based-on-machine-learning-classifiers\/297032<\/a><\/p>\n https:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.142.4092&rep=rep1&type=pdf<\/a><\/p>\n![\"\"](\"https:\/\/www.alpha-quantum.com\/blog\/wp-content\/uploads\/2022\/06\/img_62bad0a2e3527.png\")
<\/a><\/p>\nGoals of this article<\/h3>\n
\n
\n<\/strong><\/p>\n\n
\n
Offline databases or real-time, live full URL-path categorization<\/h3>\n
Other use cases of URL categorization<\/h3>\n