Frequently Asked Question

Data Pipes is a cloud-native data management platform that helps organizations quickly become data-driven by creating a shared repository of trusted data, allowing anyone in the company to start analyzing data, while ensuring the data is well-governed. We help simplify and streamline data management for your teams, so you can focus on extracting value from your data. Data Ingest With a no code interface, Data Pipes users can create data ingestion pipelines from a variety of data source connectors such as SQL databases, CSV, IoT and REST API. Data Pipes supports Change Data Capture from RDBMS, data is perfectly replicated in minutes, with minimal load on the source systems. During Ingestion, Data Pipes provides direct integration with cloud hosted data preparation tools for users to clean, normalize and prepare data from an intuitive user interface. In addition, data containing Personally Identifiable Information (PII) is automatically identified and users can select how data containing PII can be masked, tokenized or tagged. Data Catalog At the core of Data Pipes is the Data Catalog that provides users with a shared repository of trusted data. Searchable like Google, it allows users to discover and search for existing data, create and edit any metadata from the datasets. Data Pipes also features an automated Data Lineage that helps the users understand where the data is coming from and how it is being used, to validate that they are working with the right data. With direct integration with cloud hosted analytical tools, Data Pipes allows users to go directly from the catalog to analyzing data, machine learning modeling and sharing them across the organization. Data Governance Data Pipes provides a powerful suite of tools to control access to data for users. Authorized users can create rules that grant users access to data based on their Role and Department, these rules can grant or deny access to specific tables, and individual columns and rows. Individual columns in Data Pipes can be marked as classified, this allows rules to be easily created to deny access for unauthorized users to classified data. Tables containing classified data can also be completely removed from the Catalog. In addition, Data Pipes automatically keeps an audit trail of data access and permission changes by the users in real-time. These audit trails can be easily exported for any audit purposes.

Data Pipes works with expertise partners for custom requirements that may not be in our upcoming roadmap. Speak to one of our product specialists and indicate any special requirements you may need here: datapipes.io/contact

Request for a personalized demo with one of our experts here: datapipes.io/contact

With our expertise working with large enterprises on highly scalable data management platforms and data modeling to meet their needs while adhering to data privacy and security requirements, Data Pipes has seen success in multiple Institutes of High Learnings, Smart Cities and the Public Sector. Here's one of our recent successes with the SMART Urban Co-Innovation Lab in Singapore

There are three primary users for Data Pipes Data Office Administrators Data Office Administrators manage the Data Pipes application. They connect the various data sources to the data lake and organize those datasets in Domains. Data Office Administrators have access to necessary log files to ensure compliance with data security requirements. Administrators have the ability to create new domains and assign domain owners, create new ingestion pipelines, view access permission rules and export the logs, and manage features for users or a set of users. Domain Owners Domain Owners manage access control and data quality for a set of tables. Domain owners typically have some expertise in the IT systems providing the data, which allows them to provide the right context. A typical Domain Owner would be appointed by the head of an operational department (ie. Finance, HR) or be part of a Data Office. Domain Owners have the ability to document tables to share domain expertise, create access control rules to control who can access their data, and answer to access requests with the option to set a time limit for access. Data Users Data Users create data assets used - dashboards, predictive models, etc. Data Pipes provide them with a tightly integrated analytics workbench where they can discover data, validate which dataset to use, request access if necessary, perform data cleaning and preparation, and open the data in typical analytical tools.

Data Pipes users can easily create ingestion pipelines from their selected data source connectors from a no code interface. Data Pipes supports CDC (Change Data Capture) technology to perfectly replicate data from data sources into the data lake.

Data Pipes comes with an extensive list of data source connectors for on-premises and cloud databases, SaaS applications, CSV, NoSQL sources, REST API and more to quickly load data into your cloud environment.

Before the data reaches the data lake during Ingest, Data Pipes connects with multiple cloud-hosted data transformation tools to clean, join and aggregate data from an easy-to-use visual interface. Data users can choose to write custom code or select from over 250 pre-built drag and drop transformations like data normalization, filtering anomalies, converting data to standard formats, correcting invalid values, joining with other datasets, finding duplicates and adding/removing columns.

Data Pipes also allows transformation to be done after the data is loaded into the data lake. Organizations can transform their raw data at any time, when necessary, and not a compulsory step in the data pipeline. This provides faster time to value for business intelligence teams and allows for flexibility for data teams to transform data whenever they need to.

During Ingestion, data containing Personally Identifiable Information (PII) such as full names, home addresses, contact information, social security numbers are automatically identified by Data Pipes. The user ingesting the data can select how data containing PII can be masked, tokenized or tagged before it reaches the data lake.

We can sync data into Data Pipes every 5minutes, every hour, every day, or even every week. If you have any special requirements for your business, speak to one of our product specialists here: datapipes.io/contact

If you’re looking for where your dataset or table is, Data Pipes features a search bar at the top of the page. Using the search can take you quickly to your dataset, you can filter the search results to specific domains within the organization. Data Pipes also allows you to favorite and tag tables so that you can easily access them from the Catalog.

Each table will belong to an owner, which is shown on the table information page in the catalog. By default, the owner can change the following options in the catalog for each table, all these metadata are searchable from the Catalog. Table tags: Table tags are used to categorize tables, and can be used by users to search for tables Table & Columns aliases: Table names and columns names can be aliased, to have user friendly names, without changing the underlying data structure Table description: A general description of what an end user should expect to find in the table Columns description: For each column, a free text field to explain any specificities Column Security: For each column, a dropdown that allows data owners to select the access level users need to see that column and access its data. Column Sensitivity: For each column, a dropdown that allows to select the sensitivity of the data. Table owner: Who’s the user responsible for maintaining the metadata and granting access to the data. Row Representation: Description of what a single row represents.

From the table information page, users who are granted access to the table can consume the data directly from the catalog using visualization and machine learning tools. Data Pipes supports all the popular business intelligence, data science and machine learning tools in the market as long as they are JDBC compliant. Get in touch with us to find out whether your favorite tool is supported by Data Pipes here: datapipes.io/contact

Domain Owners manage access control to tables in their own domain. With the governance feature of Data Pipes, domain owners can define roles for users in their domain, and choose the datasets they can access. When a new user is created, they can also be assigned to one of the roles. Data users can also request access to the table directly from the catalog interface, the domain owners will have to grant the access for the users to consume the data using BI tools

Data Pipes provides users with a suite of Data Governance tools to control access to their data. Columns within the table can be tagged by Table Owners with classifications to restrict access for users to columns. Rules can be created on top of these classifications to perform column level security. Roles can be created and assigned to users by Domain Owners, and these rules can control access to tables, and columns. Get in touch with us to learn more about Data Pipes can help you govern your data: datapipes.io/contact

Data Pipes automatically keeps an audit trail of data access and permission changes by the users in real-time. These audit trails can be easily exported for any audit purposes.

Data Pipes provides an annual subscription model for its software. It will be fully deployed in your cloud, so that you have the assurance and fully control that your data is secured at every step of the way.

We’re very serious about security. Today, we are ISO27001 certified, meaning that we adopt the information security standards and ensure that security is part of our company’s culture, ensuring resilience from cyberthreats.

Yes, specific tables and columns can be blocked, masked and tokenized from being replicated into the Data Lake. This helps protect the personally identifiable information (PII) in the data.

No. Data Pipes does not perform scanning of the data files for malware or virus. However, the data in the file (e.g. excel, pdf, etc) will be extracted and stored into the Data Lake. Through this process of flattening the files, no malicious codes will be stored into the Data Lake.

The data moving from your sources to Data Pipes is encrypted at rest and in transit using industry-standard cryptographic protocols (TLS 1.2+). Attempts to connect over an unencrypted channel will be redirected to an encrypted channel.

All Data Pipes Platform subscriptions include Enterprise Support that provides easy access to highly skilled technical resources with prompt response times to receive the assistance you need. Data Pipes is committed to providing high-quality support and resolutions to its customers based on the response time guidelines indicated in our Response Time Table. Any bugs identified in Data Pipes will be logged by the Data Pipes Support team, who will coordinate with the development team to address the issue.

With Enterprise Support, you get 24x7 technical support from high-quality engineers and a designated Customer Success Manager (CSM) to coordinate access to subject matter experts. 24x7 access to Support Engineers via Email, Slack, Ticketing Portal and Phone Support. You can have an unlimited number of contacts that can open an unlimited amount of cases. Response times for general guidance is less than 24 hours, system impaired is less than 12 hours, production system impaired is less than 4 hours, production system down is less than an hour, and business critical system down is less than 15 minutes. The Enterprise Support offers local support resources with a toll-free local support contact for customers. Enterprise Support No. of Ticket Submission: Unlimited Technical support business hours: 24x7 Phone Contact : Available Assigned Customer Success Manager : Available Response time for unscheduled software maintenance: 24 hours General usage question (Priority 5): 2 business days for response Non-production system impaired (Priority 4): 1 business day for response. Resolution in 5 business days Feature impaired response time (Priority 3): 2 business days response time Product system impaired response time (Priority 2): 1 business day response time Business critical system response time (Priority 1): 4 hours response time

A Customer Success Manager (CSM) will be your dedicated technical point of contact who helps you onboard, provides advocacy and guidance using best practices, coordinates access to subject matter experts, assists with case management, presents insights and recommendations. Quickly and efficiently get answers to any Data Pipes inquiries via the designated Customer Success Manager (CSM) who is familiar with you and your account.

Data Pipes configures Sentry, an Application Monitoring tool to monitor the performance and track errors. This informs the Data Pipes teams of any time-outs and errors, as well as performance issues to quickly resolve performance issues.