Databricks Data Engineering

Name: Databricks Data Engineering
Brand: Databricks
Rating: 4.6 (793 reviews)

Databricks Data Engineering is a managed set of cloud services for building, running, and monitoring batch and streaming data pipelines on a lakehouse, offering ingestion connectors, declarative Spark pipeline authoring, and orchestration across AWS, Azure, and GCP.

Est 2013|Valuation $188B|Raised $33.1B

LinkedIn 1.3M|Website|Web traffic 4.3M|FTEs 16,133|HQ San Francisco, California, US

MarketsData Warehouses & Lakehouses Data Integration (ETL/ELT)

Published May 16, 2026

01Common complaints

1) An organization seeks warehouse simplicity for BI-first workloads - If most workloads are SQL dashboards and ad hoc analytics with many casual users, a warehouse-first product may be easier to manage and predict. 2) The organization cannot staff platform engineering and governance functions properly - Community feedback indicates substantial effort may be required around networking, identity, CI/CD, cluster policies, and governance to run Databricks efficiently. 3) The organization requires strictly predictable, fixed monthly pricing - Consumption-based DBU plus cloud infrastructure costs can vary. If the organization cannot tolerate variability, consider capacity-reserved models or negotiated fixed commitments and careful cost modeling. 4) The team is primarily non-programmers - Reviewer feedback notes the programming requirement (Python/SQL/Scala) can be a barrier for teams seeking low-code/no-code solutions. 5) The organization expects serverless operation without cost tradeoffs - Multiple users report serverless SQL can be materially more expensive than classic compute options in some scenarios.

Deal disqualifiers (poor fit)

Teams without Spark or engineering skills and no appetite to build platform practices.
Organizations seeking primarily point-and-click ETL with minimal code.
Organizations needing strictly predictable, fixed monthly pricing without consumption variability unless addressed through committed or custom pricing arrangements.

Limitations acknowledged by documentation and product posture

Some features are explicitly labeled Preview or Public Preview, such as the Data Engineering Agent (Feb 4, 2026 documentation). Preview status can imply potential breaking changes, limited SLA/support guarantees, and uneven regional/cloud availability.

Common limitations from user discussions

Cost surprises, particularly for serverless SQL and specific connector/storage behaviors.
Cloud parity gaps: features not uniformly available on AWS, Azure, and GCP at the same time.
Operational friction: cluster startup times and the need for careful platform configuration.

Roadmap visibility

Databricks announces roadmap items through events (Data + AI Summit) and documentation updates, but exact GA dates for preview features may not be firm.

02Why they buy

Unique selling points supported by user sentiment and product documentation

Engineering-centric flexibility: strong Spark-based execution and code workflows.
Multi-cloud availability: AWS, Azure, and GCP deployments.
Governance capabilities: Unity Catalog provides centralized access control, auditing, and lineage across workspaces as described in public documentation.

Where competitors may have advantage

Snowflake: perceived simplicity for SQL analytics and a managed warehouse experience.
Microsoft Fabric: integrated suite and Power BI alignment for Microsoft-centric environments.

03Field reports

This section mixes review-platform sentiment with community discussions. Review platforms can skew toward larger customers and may have selection bias.

Review platforms

G2 (Databricks product reviews): - G2 shows Databricks rated highly with a large number of reviews (as displayed on the page captured). - Example negative limitation (paraphrased from a G2 reviewer): one reviewer noted a learning curve when users lack experience with Python, SQL, or Scala, making the platform difficult for less technical users. - Example positive aspect (paraphrased from a G2 reviewer): reviewers cited ease of integration with major cloud providers such as Azure, AWS, and Google Cloud. TrustRadius (Databricks Data Intelligence Platform): - TrustRadius lists a modest number of reviews and ratings for the broader Databricks suite; detailed quotes were not captured in this research run. Gartner Peer Insights: direct access to user quotes is often restricted and may require login; this run did not capture specific Gartner Peer Insights quotes.

Community / social (Reddit, Hacker News)

Community discussions often surface operational pain points (cost, bugs, rollout complexity) more prominently than review platforms. Reddit: working with Databricks vs Snowflake (r/dataengineering): - A thread provides cost and platform engineering perspectives. One user summarized that to achieve cost advantages with Databricks, teams may need to build a data platform around it, which can be extensive. - The same thread mentions setup complexity (Azure-focused) around networking, endpoints, identity, and CI/CD, with community members describing confusion during initial configuration. - Criticism themes include features being in preview or fully supported in only one cloud, occurrence of bugs, and support responses that suggest workarounds or future features rather than immediate fixes. Reddit: Lakeflow Connect costs/storage surprises (r/databricks): - A recent thread raises questions about Lakeflow Connect storage billing being high for a moderately sized SQL Server table with frequent updates, indicating potential non-obvious cost drivers such as history retention and append-only behavior. Reddit: Serverless SQL cost premium: - A thread titled “Serverless SQL is significantly more expensive than classic—is it worth it?” highlights sticker shock; users compare SQL Pro/classic vs serverless rates. Hacker News: - A targeted search was performed for relevant discussions; this run did not capture a specific Hacker News thread snippet.

Why customers choose Databricks (common themes)

Across reviews and community discussion, the main reasons cited are:

Spark-based processing suitable for large-scale ETL and streaming.
A common environment for engineering, data science, and ML workflows.
Multi-cloud availability and ecosystem integrations.

Why some customers leave or reduce usage (common themes)

Difficulty controlling costs without strong FinOps discipline.
Platform complexity compared to fully managed warehouses.
Feature maturity inconsistency across clouds and perceived rapid change.

Vendor and vendor-adjacent materials

Databricks publishes customer stories and provides guidance materials and summit sessions (e.g., data engineering guidance, Data + AI Summit sessions). These resources are useful for understanding patterns but are marketing-oriented.

Quantified outcomes seen in community narrative

A Reddit thread references a claim that a company realized substantial cost savings by migrating from Snowflake to Databricks; this is secondhand and not independently verified in the thread itself.

Critical evaluation

- Many ROI and performance claims in the Databricks vs Snowflake space are benchmark-dependent and often vendor-influenced. - Example: competitor-authored pages may present comparative performance claims; such material should be treated as marketing unless the methodology and raw data are reviewed. - For procurement-grade evaluation, buyers should run: - workload-specific proofs of concept (ETL, streaming, BI concurrency) - cost replay using a production-like job mix - governance/security capability mapping (Unity Catalog vs alternative controls)

04Best fit

Company size (practical ranges)

Mid-market to enterprise: typically organizations with hundreds to many thousands of employees, with dedicated data platform/engineering teams.
Smaller startups can use Databricks, but operational overhead and skills requirements often make simpler stacks more attractive unless Spark-scale is needed.

Strong-fit industries

Financial services, retail/ecommerce, media/ads tech, manufacturing/IoT, and other organizations with:
- large-scale batch processing
- streaming ingestion and transformations
- ML feature engineering pipelines

Roles/personas

Decision makers: Head of Data Engineering, Data Platform Lead, CIO/CTO, Analytics/AI leaders.
Hands-on users: data engineers, analytics engineers, ML engineers, platform engineers, and SQL analysts (particularly those already familiar with Databricks SQL).

Pain points Databricks addresses well

Scaling ETL/ELT beyond single-node systems.
Centralizing governance and lineage across data and AI assets (Unity Catalog positioning).
Handling both batch and streaming within a common processing paradigm.

05The pitch

What the product does (objective value proposition)

Databricks Data Engineering is Databricks’ set of capabilities for building, running, and monitoring batch and streaming data pipelines on the Databricks platform (the “Data Intelligence Platform” / lakehouse). In recent years, Databricks’ data engineering positioning increasingly centers on Lakeflow as a comprehensive set of data engineering capabilities, spanning ingestion, transformations (including declarative pipelines), and orchestration/jobs. Databricks provides these features as a managed service on AWS, Azure, and Google Cloud. Key components emphasized in current documentation include Lakeflow Connect (ingestion connectors), Lakeflow Spark Declarative Pipelines (SDP) (pipeline authoring/execution), and Lakeflow Jobs (orchestration). (Official vendor documentation.)

Target market and market positioning

- Primary buyers/users: data engineering teams, analytics engineering teams, platform data teams, and organizations standardizing on Spark/Delta Lake. - Positioning: a platform that integrates ETL, streaming, governance, analytics, and ML/AI on a lakehouse architecture; developer-centric with notebooks and code-first workflows. - Common adoption patterns (observed in user discussions): - Organizations often retain Databricks for Spark-heavy ETL, streaming, and ML, while using a separate warehouse (often Snowflake) for BI—though Databricks SQL/serverless is frequently evaluated as a consolidation path.

Get a comprehensive analysis of Databricks Data Engineering including market position, competitive landscape, adoption trends, and peer benchmarks.

Download full report →

Published May 16, 2026

01Common complaints

An organization seeks warehouse simplicity for BI-first workloads
- If most workloads are SQL dashboards and ad hoc analytics with many casual users, a warehouse-first product may be easier to manage and predict.
The organization cannot staff platform engineering and governance functions properly
- Community feedback indicates substantial effort may be required around networking, identity, CI/CD, cluster policies, and governance to run Databricks efficiently.
The organization requires strictly predictable, fixed monthly pricing
- Consumption-based DBU plus cloud infrastructure costs can vary. If the organization cannot tolerate variability, consider capacity-reserved models or negotiated fixed commitments and careful cost modeling.
The team is primarily non-programmers
- Reviewer feedback notes the programming requirement (Python/SQL/Scala) can be a barrier for teams seeking low-code/no-code solutions.
The organization expects serverless operation without cost tradeoffs
- Multiple users report serverless SQL can be materially more expensive than classic compute options in some scenarios.

Deal disqualifiers (poor fit)

Teams without Spark or engineering skills and no appetite to build platform practices.
Organizations seeking primarily point-and-click ETL with minimal code.
Organizations needing strictly predictable, fixed monthly pricing without consumption variability unless addressed through committed or custom pricing arrangements.

Limitations acknowledged by documentation and product posture

Some features are explicitly labeled Preview or Public Preview, such as the Data Engineering Agent (Feb 4, 2026 documentation). Preview status can imply potential breaking changes, limited SLA/support guarantees, and uneven regional/cloud availability.

Common limitations from user discussions

Cost surprises, particularly for serverless SQL and specific connector/storage behaviors.
Cloud parity gaps: features not uniformly available on AWS, Azure, and GCP at the same time.
Operational friction: cluster startup times and the need for careful platform configuration.

Roadmap visibility

Databricks announces roadmap items through events (Data + AI Summit) and documentation updates, but exact GA dates for preview features may not be firm.

02Why they buy

Unique selling points supported by user sentiment and product documentation

Engineering-centric flexibility: strong Spark-based execution and code workflows.
Multi-cloud availability: AWS, Azure, and GCP deployments.
Governance capabilities: Unity Catalog provides centralized access control, auditing, and lineage across workspaces as described in public documentation.

Where competitors may have advantage

Snowflake: perceived simplicity for SQL analytics and a managed warehouse experience.
Microsoft Fabric: integrated suite and Power BI alignment for Microsoft-centric environments.

03Field reports

This section mixes review-platform sentiment with community discussions. Review platforms can skew toward larger customers and may have selection bias.

Review platforms

G2 (Databricks product reviews):

G2 shows Databricks rated highly with a large number of reviews (as displayed on the page captured).
Example negative limitation (paraphrased from a G2 reviewer): one reviewer noted a learning curve when users lack experience with Python, SQL, or Scala, making the platform difficult for less technical users.
Example positive aspect (paraphrased from a G2 reviewer): reviewers cited ease of integration with major cloud providers such as Azure, AWS, and Google Cloud.

TrustRadius (Databricks Data Intelligence Platform):

TrustRadius lists a modest number of reviews and ratings for the broader Databricks suite; detailed quotes were not captured in this research run.

Gartner Peer Insights: direct access to user quotes is often restricted and may require login; this run did not capture specific Gartner Peer Insights quotes.

Community / social (Reddit, Hacker News)

Community discussions often surface operational pain points (cost, bugs, rollout complexity) more prominently than review platforms.

Reddit: working with Databricks vs Snowflake (r/dataengineering):

A thread provides cost and platform engineering perspectives. One user summarized that to achieve cost advantages with Databricks, teams may need to build a data platform around it, which can be extensive.
The same thread mentions setup complexity (Azure-focused) around networking, endpoints, identity, and CI/CD, with community members describing confusion during initial configuration.
Criticism themes include features being in preview or fully supported in only one cloud, occurrence of bugs, and support responses that suggest workarounds or future features rather than immediate fixes.

Reddit: Lakeflow Connect costs/storage surprises (r/databricks):

A recent thread raises questions about Lakeflow Connect storage billing being high for a moderately sized SQL Server table with frequent updates, indicating potential non-obvious cost drivers such as history retention and append-only behavior.

Reddit: Serverless SQL cost premium:

A thread titled “Serverless SQL is significantly more expensive than classic—is it worth it?” highlights sticker shock; users compare SQL Pro/classic vs serverless rates.

Hacker News:

A targeted search was performed for relevant discussions; this run did not capture a specific Hacker News thread snippet.

Why customers choose Databricks (common themes)

Across reviews and community discussion, the main reasons cited are:

Spark-based processing suitable for large-scale ETL and streaming.
A common environment for engineering, data science, and ML workflows.
Multi-cloud availability and ecosystem integrations.

Why some customers leave or reduce usage (common themes)

Difficulty controlling costs without strong FinOps discipline.
Platform complexity compared to fully managed warehouses.
Feature maturity inconsistency across clouds and perceived rapid change.

Vendor and vendor-adjacent materials

Quantified outcomes seen in community narrative

A Reddit thread references a claim that a company realized substantial cost savings by migrating from Snowflake to Databricks; this is secondhand and not independently verified in the thread itself.

Critical evaluation

Many ROI and performance claims in the Databricks vs Snowflake space are benchmark-dependent and often vendor-influenced.
- Example: competitor-authored pages may present comparative performance claims; such material should be treated as marketing unless the methodology and raw data are reviewed.
For procurement-grade evaluation, buyers should run:
- workload-specific proofs of concept (ETL, streaming, BI concurrency)
- cost replay using a production-like job mix
- governance/security capability mapping (Unity Catalog vs alternative controls)

04Best fit

Company size (practical ranges)

Mid-market to enterprise: typically organizations with hundreds to many thousands of employees, with dedicated data platform/engineering teams.
Smaller startups can use Databricks, but operational overhead and skills requirements often make simpler stacks more attractive unless Spark-scale is needed.

Strong-fit industries

Financial services, retail/ecommerce, media/ads tech, manufacturing/IoT, and other organizations with:
- large-scale batch processing
- streaming ingestion and transformations
- ML feature engineering pipelines

Roles/personas

Decision makers: Head of Data Engineering, Data Platform Lead, CIO/CTO, Analytics/AI leaders.
Hands-on users: data engineers, analytics engineers, ML engineers, platform engineers, and SQL analysts (particularly those already familiar with Databricks SQL).

Pain points Databricks addresses well

Scaling ETL/ELT beyond single-node systems.
Centralizing governance and lineage across data and AI assets (Unity Catalog positioning).
Handling both batch and streaming within a common processing paradigm.

05The pitch

What the product does (objective value proposition)

Target market and market positioning

Primary buyers/users: data engineering teams, analytics engineering teams, platform data teams, and organizations standardizing on Spark/Delta Lake.
Positioning: a platform that integrates ETL, streaming, governance, analytics, and ML/AI on a lakehouse architecture; developer-centric with notebooks and code-first workflows.
Common adoption patterns (observed in user discussions):
- Organizations often retain Databricks for Spark-heavy ETL, streaming, and ML, while using a separate warehouse (often Snowflake) for BI—though Databricks SQL/serverless is frequently evaluated as a consolidation path.

Get a comprehensive analysis of Databricks Data Engineering including market position, competitive landscape, adoption trends, and peer benchmarks.

Download full report →