The way information on the internet is consumed is changing. It's shifting from humans searching pre-crawled information on Google via point and click to AI agents doing real-time targeted crawling from sources of truth. The interface, the workflow, and the density of retrieved data and its fidelity, which worked for humans, doesn't work for AI agents.

At Crustdata, we are building the gateway to the internet for AI agents. In simple terms, we're building the APIs for AI Agents to access real-time data from sources of truth. We already serve dozens of enterprise customers, are profitable and growing very fast. We're backed by some of the best investors in Silicon Valley including Y Combinator, General Catalyst, SV Angel, A Capital and Liquid 2 Ventures among others.

Mission

At Crustdata, we are building the gateway to the internet for AI agents. Essentially we are building Google search as an API where the data isn’t fetched from the pre-crawled index but rather crawled from the sources of truth, on-demand and real-time.

We started small, focusing on Sales and Marketing AI agents. Now we are expanding to more sources and additional markets.

What you’ll be doing

This is a senior engineering role. You will be directly responsible for designing, developing and owning core parts of our stack that includes:

• API interfaces
• Generic and scalable crawlers
• Data platform
• and much more

This is a high agency role with lot of ambiguity as a part of early stage startup. You will be reporting directly to the CEO. You will have direct contribution in the future trajectory of our business and the speed at which we are able to realize our vision.

Qualifications

• 5+ years experience developing Python and C/C++ software for production systems
• Experience with designing and maintaining production data warehouse systems
• Strong Linux fundamentals

What makes you stand out

• Experience building custom web crawlers
• Experience in reverse engineering software or hardware systems
• Great taste for writing simple code