Data Apps describes the new wave of cloud data warehouse (CDW) consumers that build applications directly on top of the CDW. From practitioners at data teams to VCs and founders, the nascent movement has its proponents and critics, and a growing list of new startups.
The advantages are clear: data is central and commonplace. Data structures become similar or identical when it comes from the same service, and the knowledge about how to exploit them is the same. Still, the new wave of tools goes beyond how to exploit knowledge.
Data apps are powering internal tools, APIs, embedded analytics, automation, and more. State-of-the-art CDWs are powerful but they are not built to be an OLTP database. Because of this, data apps built directly on CDWs must bring their own sophisticated caching layers, and still suffer from performance issues like high latency and stale data due to the inherent batch nature of ingestion. There is also a lack of support for transactions, difficulty doing lookup queries, and an inability to efficiently handle constraints due to foreign key limitations. These tradeoffs are well explained by Hightouch.
As an added side effect, when you move data consumption from the Business Intelligence (BI) tool and into data apps, your users start to lose the context that the data is coming from a warehouse. They see the data in a web interface, a support tool, an app, and it’s often right next to primary application data, data that is coming from the primary transactional database. This makes stale data more of a problem.
Data apps shift the pressure to evolve from the BI tool back to the data warehouse:
Now the Data Warehouse has the heat in the stack
For teams wanting to build new, performant, reactive applications where data correctness and freshness are crucial, Materialize serves as a natural building block.
If the business is starting from scratch without a CDW but wants to employ second-wave tools without the downsides of stale data, then Materialize can serve as the data store and form the backbone of an active data stack. Immediate benefits include:
- Fresh Data - data applications built on the CDW are inherently stale because updates rely on batch processing while Materialize relies on streaming processing
- Fast and Responsive - incrementally maintained materialized views provide real time data and tail provides a powerful mechanism to drive events downstream
- Postgres-compatibility and SQL - Materialize presents as a Postgres-compatible database and its primary user experience is Postgres-compatible SQL
Using Materialize doesn’t mean throwing away the BI tool either, as of June 2022, Materialize PG Compatibility enables it to integrate with Metabase and it has alpha integrations with Superset, Preset, Looker and Google Data Studio.
Materialize as a heat sink for the stack
Focus on what matters
Businesses know that data is the new oil, but only a few can translate that oil into insights or value beyond their business’ core knowledge. Data apps go a step further than BI tools in helping teams leverage their data to power a variety of use cases like:
- Internal tools (Appsmith, Bubble and Retool)
- Automation (Hightouch, Census)
- Sales (Calixa, Endgame, Pocus)
- Security (Hunters, Panther)
- Notebooks (Hex), visualizations (Preset), or headless BI (CubeJS)
Each vertical contains knowledge with a high chance of being unique or missing for a growing business. Developing a successful sales workflow or a smooth heuristic to predict Product Qualified Leads (PQLs) can take months while a product can enable similar knowledge over the same data in a few clicks. Researching, aggregating, and joining multiple data sources to detect a security incident is a task in which even finding the right people for the job is hard. A data app can enable the same knowledge or even a more appropriate process in less time.
While these products focus on getting insights, organizations can focus on what matters: their core business.
Stacks
The word architecture sounds archaic nowadays. Stacks are getting more flexible and integrations are smoother than ever. Adding or swapping a component is much simpler than it was years ago when the architecture word was still predominant.
In some stack scenarios, it’s possible to combine multiple data apps: Embedding a Hex dashboard into Retool consuming data from CubeJS.
Combining multiple data apps
I’m overcomplicating the case, but you can get the point.
No-code or low-code features are also present in data apps, laying or sharing part of the landscape. Every step inside these products tries to make work simpler.
And Materialize isn’t an exception. Processing and serving the most up-to-date information is difficult and stressful, and that’s where Materialize can alleviate the pain. As a SQL partisan technology, it abstracts streaming techniques to help generate up-to-date insights in just a few lines of code.
Building a new point of consumption for new insights with fresh data should be as simple as adding these tools to the stack.
Multiple data apps reading from Materialize
Hands-on
You can go from set-up to results in less than ten minutes.
After signing up for a data app, like Hex, the next step is to integrate it with Materialize. The integration is done through a single step using the Postgres-compatible connection.
Adding a data connection in Hex
If you have been a long-time user of Materialize, you will notice one detail. It’s using a username and password! One of the many new features available in the cloud. Alternatively, the significant majority of data apps also let you operate with certificates.
The next step in data apps tends to be running a query and letting the product do the rest, or setting up a particular table or materialized view to consume. In the Hex case, it is to create a project and run a SQL query or a custom Python script to read data from Materialize.
Running a query in Hex
Data apps can provide a fast and simple way to iterate and accomplish a variety of use cases while adding Materialize reduces billing costs, improves performance, and stays up to date with the latest market trends. To see this in action with Materialize, Hex, and Redpanda, check out Marta’s Hex application for Data Council Austin!
If you’re interested in digging deeper into Materialize to see if it can serve as the engine to your own data apps, try Materialize locally using our docs, join us in the Community or set up a demo with our team.
