Need to have audit and compliance oversight on Data Highways

Data is at the heart of developing new services to improve user or customer experience. Whilst it has been so fragmented and rarely accessible whilst “in motion”, organisations have created data highways with technologies such as Apache Kafka to centralise transport and facilitate data democratisation across applications and analytics solutions.

Photo by Annie Spratt on Unsplash

For example, a payment fraud service notifying a customer by SMS of a double transaction by the same merchant before he has left the shop.

Developing such a service that may give the credit card issuer a competitive advantage and can be completely decoupled from the (different independent) payment services themselves. Such a service could be released to market with so much more ease if developed on a Data Highway. Kafka Streams is a great technology for these sorts of usecases.

Providing autonomy and removing as much friction as possible such as this in order to develop new services is essential to gain market share. 

Which causes nightmares for risk and compliance teams as well as regulators.

Fintechs are a great example, they were synonymous with modern development practices and new digital services. They had it easy initially, able to trade in a sort of no-man’s land from a regulatory perspective and were able to take big chunks out of the financial services markets.

In order to continue give product teams this freedom, risk and compliance controls need to be reported and enforceable increasingly on the fly. Data highways provide a huge upside but also an enormous risk if not monitored and governed correctly.

Photo by Basil Samuel Lade on Unsplash

New solutions that allows compliance officers to see who is accessing what data, what applications are consuming and producing data and to apply governance controls on the data highways without restricting or slowing down product teams are needed.  

Share your thoughts by submitting comments!