Web Scrape Developer (Python)

At ShopGrok, we use unique market insights and best in class data analytics to help retailers and consumer product manufacturers make better product and pricing decisions.

We enable retail buyers and pricing teams to get real time, actionable insights so that they can use more science and less human judgement in their decision making and strategy formulation. We track 10s of millions of price points every week, have a growing list of large, household name retailers who access our data every day, and we’re growing quickly.

At ShopGrok, data is at the heart of what we do. Your main responsibilities will be working closely with the Founder and Senior Developers to build and deploy web crawlers, harvest data and build analytics to deploy into our data visualisation platform.

In addition to your core responsibilities of build and maintain our codebase of web crawlers, you will be curious and keen to find ways to improve our system, processes and automation. Whether its researching new crawling techniques, delving into DevOps or Data Engineering topics, or improving our documentation.

Key responsibilities:

• Analyse retailer websites to identify product and price attributes and collect data
• Build and deploy web crawlers using Python scrapy and our cloud deployment platform using best practices
• Perform web crawler scheduling, monitoring and maintenance
• Perform data processing using text matching techniques such as ‘fuzzy match’
• Advocate for exceptional code design, and contribute to shaping a culture of design at ShopGrok.
• Improve existing code base, architecture and workflows where possible.

Requirements:
• You have 2+ years experience with python either in a business environment, or through heavy exposure during your studies which can be demonstrated.
• You have a built web crawlers using python scrapy and can demonstrate them, for example via your Github profile.
• You have a very strong understanding of HTTP protocols, HTML and DOM concepts, and are adept at using XPATH to identify elements in web pages
• You have experience using Chrome Developer Tools (or equivalent) to interrogate a web page to identify page elements, data sources and API calls
• You have experience interrogating APIs and working with data in JSON format
• You have a strong knowledge of regular expressions and the python re library
• A working knowledge of SQL to analyse data. Experience building ETL workflows with tools such as Alteryx to blend, cleanse and model data is a big plus.
• You are a thoughtful and considered communicator who thrives on sharing ideas.
• You have experience using version control tools such as Git.
• Experience building and maintaining APIs using tools such as Flask would be a big plus
• Likewise, experience with DevOps in the cloud, such as maintaining virtual machines in GCP would also be a big plus

About You

• Technical: You are a strong python developer with experience building web crawlers using the python scrapy library. You're a keen learner and, where you don't know the best way to tackle an implementation problem, are happy to get your hands dirty with the rest of the team learning how to make your designs a reality.
• Hungry: You are excited for the opportunity to build out our data infrastructure, and to make your mark on a growing company.
• Collaborative: You love working with a team and are adept at explaining technical decisions to other team members, and helping them to bring their ideas to life. You are also comfortable working with other developers to ensure a consistent code base across the company.
• Thoughtful: You view software development as a never-ending process, and are always thinking about ways to improve our product and its backend infrastructure.
• Analytical: You take a hypothesis driven, data-first approach to attacking any problem, and are adept at separating a complex problem into its mutually exclusive parts and systematically finding solutions through testing and learning.
• Perceptive: You are excited by the challenge of answering complex questions in accessible ways using data, you have an eye for detail and will not rest until your implementations function as specified in the design phase.

Benefits:
• Competitive salary.
• Set the scene. As a member of a small team, you will have a dramatic impact on how we run things at ShopGrok and the growth of the company. You’ll be working with highly motivated, fast-thinking and intelligent team members from multiple walks of life.
• Flexible work arrangements - our founder and team typically work from home approx ~50% of the time