New Momentum LLC.

Extract Transform and Load with Analytic Portal


As an Angel funded organization, New Momentum needed to quickly create a product that could serve 2 different types of customers from a universal user interface. CTO Services was contracted to be the development team in the initial stage of existence to be able to generate revenue more quickly. We assembled a  team of seven consultants and began to create a portal user interface with 20 portlets to provide analytical information and alerts to end users to find the needles in haystacks of electronic component data.

The “haystacks” were just that. The first haystack was several years of thousands of emails per day that needed to be parsed to collect electronic component parts data being sold in what is known as the “Gray Market” for electronic components. Once the data was extracted from the emails it had to be transformed and normalized into structured data that could be stored in a database for further processing. Company information had to be cross linked to aliases of the same company as well as quantities and prices to a massive cross reference database of electronic components in an attempt to identify companies selling counterfeit products, selling unauthorized products, or selling electronic products at prices below contractual agreements so that those distributors could be contacted for remediation or possible legal action.

The second haystack was developing a web crawler that could fake end user level searches of the largest internet-based component distributors and extract all of the electronic components. From the 15 companies that data was being extracted, roughly 1 to 2 million records were collected, transformed with cross referenced masters of manufacturer, manufacturer part number, and quantity price breaks. Once the data was extracted into a temporary database it was then loaded into a data warehouse so that the analytical portals could provide relevant information to the clients of the service.

The nearly 500 Gigabytes of data generated each month was pushing the limits of available hardware so a secondary effort summarized 32 day old data to a data warehouse with 75% reduction in space consumed for that day and then that source data was purged each day.

After roughly one year initial customers signed on to the subscription service and we helped to hire and train a development team to carry the product forward and ultimately to the acquisition by Market Track.