Aplicando técnicas de web scraping e web crawling para comparar preços de produtos de E-commerces
Carregando...
Arquivos
Data
2023-06
Autores
Orientador
Ahlert, Edson Moacir
Banca
Pretto, Fabrício
Dertzbacher, Juliano
Título do periódico
ISSN
Título do Volume
Editor
Resumo
Em meio a inúmeros e-commerces, com produtos vendidos por numerosos vendedores, além da própria plataforma, fica difícil para o consumidor encontrar o melhor preço para o produto que está buscando, exigindo uma árdua pesquisa em vários sites, por meio de uma infinda quantidade de cliques. Este estudo teve como objetivo desenvolver um comparador de preços de produtos de e-commerces, utilizando técnicas de web crawling e web scraping, incorporando a funcionalidade de cashback para impulsionar a economia do consumidor. O foco foi na navegação entre páginas, extração, tratamento, armazenamento e disponibilização das informações. O framework Scrapy foi utilizado para a aplicação destas técnicas. O Scrapy faz requisições aos quatro principais e-commerces definidos pelo autor, resgatando os dados de cada um e armazenando-os localmente em um arquivo CSV. Em seguida, é feita uma requisição ao site comparemania.com.br para resgatar as três melhores ofertas de cashback e as respectivas plataformas, para cada e-commerce. O cálculo do cashback sobre o valor final do produto de cada e-commerce é realizado, e as informações são apresentadas em uma planilha, incluindo informações como, principalmente: o e-commerce anunciante, o vendedor, o valor final, o cashback (de três plataformas) e o valor final com cashback. A obtenção dos dados para cada e-commerce leva 15 segundos, sendo disparadas até 4 requisições no máximo. Os resultados obtidos evidenciaram a importância de aplicar corretamente os fundamentos em cada etapa do projeto. A elaboração e teste das spiders foram cruciais para o aperfeiçoamento do projeto, permitindo ajustes, correções e análises detalhadas da lógica do código. Cada e-commerce apresentou suas peculiaridades, exigindo uma análise cuidadosa do HTML, seleção precisa dos elementos e atributos e desenvolvimento de um código XPath consistente e resistente a modificações. Após um extenso processo de testes, as spiders foram consolidadas, realizando com precisão as requisições, extração de informações e funcionando eficientemente.
In the midst of numerous e-commerces, with products sold by numerous sellers, in addition to the platform itself, it is difficult for the consumer to find the best price for the product he is looking for, requiring an arduous search on several sites, through an endless amount of clicks. This study aimed to develop a price comparison tool for e-commerce products, using web crawling and web scraping techniques, incorporating the cashback functionality to boost the consumer's economy. The focus was on navigation between pages, extraction, treatment, storage and availability of information. The Scrapy framework was used to apply these techniques. Scrapy makes requests to the four main e-commerces defined by the author, retrieving data from each one and storing them locally in a CSV file. Then, a request is made to the comparemania.com.br website to redeem the three best cashback offers and the respective platforms, for each e-commerce. The calculation of the cashback on the final value of the product of each e-commerce is carried out, and the information is presented in a spreadsheet, including information such as, mainly: the advertiser e-commerce, the seller, the final value, the cashback (from three platforms) and the final value with cashback. Obtaining data for each e-commerce takes 15 seconds, triggering up to 4 requests at most. The results obtained showed the importance of correctly applying the fundamentals in each stage of the project. The elaboration and testing of the spiders were crucial for the improvement of the project, allowing adjustments, corrections and detailed analysis of the code's logic. Each e-commerce had its peculiarities, requiring careful analysis of the HTML, precise selection of elements and attributes, and development of a consistent XPath code that is resistant to modifications. After an extensive testing process, the spiders were consolidated, accurately performing requests, extracting information and working efficiently.
In the midst of numerous e-commerces, with products sold by numerous sellers, in addition to the platform itself, it is difficult for the consumer to find the best price for the product he is looking for, requiring an arduous search on several sites, through an endless amount of clicks. This study aimed to develop a price comparison tool for e-commerce products, using web crawling and web scraping techniques, incorporating the cashback functionality to boost the consumer's economy. The focus was on navigation between pages, extraction, treatment, storage and availability of information. The Scrapy framework was used to apply these techniques. Scrapy makes requests to the four main e-commerces defined by the author, retrieving data from each one and storing them locally in a CSV file. Then, a request is made to the comparemania.com.br website to redeem the three best cashback offers and the respective platforms, for each e-commerce. The calculation of the cashback on the final value of the product of each e-commerce is carried out, and the information is presented in a spreadsheet, including information such as, mainly: the advertiser e-commerce, the seller, the final value, the cashback (from three platforms) and the final value with cashback. Obtaining data for each e-commerce takes 15 seconds, triggering up to 4 requests at most. The results obtained showed the importance of correctly applying the fundamentals in each stage of the project. The elaboration and testing of the spiders were crucial for the improvement of the project, allowing adjustments, corrections and detailed analysis of the code's logic. Each e-commerce had its peculiarities, requiring careful analysis of the HTML, precise selection of elements and attributes, and development of a consistent XPath code that is resistant to modifications. After an extensive testing process, the spiders were consolidated, accurately performing requests, extracting information and working efficiently.
Descrição
Palavras-chave
Comparador de sites; Web scraping; E-commerce; Site comparator
Citação
THUMS, Bruno Brill. Aplicando técnicas de web scraping e web crawling para comparar preços de produtos de E-commerces. 2023. Monografia (Graduação em Engenharia da Computação) – Universidade do Vale do Taquari - Univates, Lajeado, 29 jun. 2023. Disponível em: http://hdl.handle.net/10737/3556.