|By Tom Leyden||
|April 15, 2013 09:00 AM EDT||
Here is an answer I wrote to my dear friend StorageBod’s latest piece on Object Storage. As my response turned out quite a bit longer than initially intended, I thought I’d post it here as well.
Very nice blog post as usual. Let me sacrifice my first sunny afternoon of the year in Belgium to write down some of my thoughts (rather than copy/pasting a data sheet like Paul did). Oh and … NAB would have been so much better with you over there,.
Disclaimer: I am still in charge of the object storage product line at DDN. The below thoughts are my own, however, not necessarily the company’s.
“Yet, it isn’t really happening”. I expect to be punished for writing this, but Martin: you are WRONG! Object storage is happening very much. Just not necessarily out in the open. Implementing object storage is a lot more game-changing than buying a new filer from a supplier you haven’t worked with before. We would have hoped for a faster wide adoption of object storage, but if you look at other recent paradigm changes, we are on the curve: I gave my first cloud presentation back in 2007. It took about 5 years for cloud to be embraced by the enterprise market. You could say that object storage has been around for more than five years (Centera etcetera), but if you look at the recent generation of object storage and the more modern use cases, we are on schedule. I wish I could talk to you about a few of those double-digit petabyte projects I’ve been involved with in the past 2 years.
“It turns out the developers really want a POSIX file-system”. We have had the REST vs. CIFS/NFS discussion before. I believe you and I agreed that a file gateway is unavoidable in the process to wider object storage adoption. Many applications are not object-ready and while we are waiting for the ISV’s to adapt their software, an extra file system layer can do the trick. As a matter of fact, a lot of the discussions I had at NAB last week (see, here is where it helps to be at the show rather than reading 140 character synopses), both with partners and customers, showed that we are getting there: DAM and MAM players are going for direct interfaces with the object storage pool. That said, in many cases I don’t see why companies would even try object storage. Object storage was designed for large sets (multi-petabytes) of mostly immutable data. Those terabyte-scale projects to test the water, often just to replace filers, are not the right use cases. 100 TB of object storage with a file gateway on top will not be a better solution than 100TB of regular NAS.
“I’ve not gone down the route of putting in an Object storage solution because finding one which is supported across all the tools in today’s workflows is near impossible”. I agree: object storage will never be supported by all tools in your workflow. As I mentioned before, object storage was designed for data sets of mostly immutable data. As you would have seen at the DDN booth if you had visited NAB (sorry to rub it in), our object storage platform (WOS) was positioned for collaboration, distribution and archiving of media assets, but not for creation or post-processing. Those steps in the media workflow benefit from other platforms. The nice thing about DDN’s WOS is that it perfectly integrates with our other storage platforms so that each platform can be used for its best purpose.
“[…] to provide us with the sort of transparency we need to support complex digital workflows”. Maybe you should also think about making you workflows less complex?
“I regularly suggest that we put in feature requests to the tools vendors to at least support S3; the looks I generally get are one of quiet bemusement or outright hostility and mutterings about Amazon and Cloud.” I can’t imagine that you would let yourself be intimidated by that! There are plenty of examples – also in your industry – of large media deployments on S3. So it makes total sense to add these requirements. That said, deploying an internal object store probable makes more sense than using the public service.
“So give it 20 years or so and we’ll be rocking.” See? We agree. But 20 months will be more than enough for object storage to rock. Technically, if you’d count all the projects I can’t talk about, it already does.
“object-storage gateway into the base operating system” Isn’t that what WebDAV is? OSX does this just fine.
With so much going on in this space you could be forgiven for thinking you were always working with yesterday’s technologies. So much change, so quickly. What do you do if you have to build a solution from the ground up that is expected to live in the field for at least 5-10 years? This is the challenge we faced when we looked to refresh our existing 10-year-old custom hardware stack to measure the fullness of trash cans and compactors.
Aug. 25, 2016 01:15 AM EDT Reads: 1,593
This digest provides an overview of good resources that are well worth reading. We’ll be updating this page as new content becomes available, so I suggest you bookmark it. Also, expect more digests to come on different topics that make all of our IT-hearts go boom!
Aug. 25, 2016 12:45 AM EDT Reads: 4,664
The emerging Internet of Everything creates tremendous new opportunities for customer engagement and business model innovation. However, enterprises must overcome a number of critical challenges to bring these new solutions to market. In his session at @ThingsExpo, Michael Martin, CTO/CIO at nfrastructure, outlined these key challenges and recommended approaches for overcoming them to achieve speed and agility in the design, development and implementation of Internet of Everything solutions wi...
Aug. 25, 2016 12:45 AM EDT Reads: 1,898
19th Cloud Expo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterpri...
Aug. 25, 2016 12:00 AM EDT Reads: 2,956
Cloud Expo 2016 New York at the Javits Center New York was characterized by increased attendance and a new focus on operations. These were both encouraging signs for all involved in Cloud Computing and all that it touches. As Conference Chair, I work with the Cloud Expo team to structure three keynotes, numerous general sessions, and more than 150 breakout sessions along 10 tracks. Our job is to balance the state of enterprise IT today with the trends that will be commonplace tomorrow. Mobile...
Aug. 24, 2016 11:45 PM EDT Reads: 3,137
Sharding has become a popular means of achieving scalability in application architectures in which read/write data separation is not only possible, but desirable to achieve new heights of concurrency. The premise is that by splitting up read and write duties, it is possible to get better overall performance at the cost of a slight delay in consistency. That is, it takes a bit of time to replicate changes initiated by a "write" to the read-only master database. It's eventually consistent, and it'...
Aug. 24, 2016 11:30 PM EDT Reads: 3,068
To leverage Continuous Delivery, enterprises must consider impacts that span functional silos, as well as applications that touch older, slower moving components. Managing the many dependencies can cause slowdowns. See how to achieve continuous delivery in the enterprise.
Aug. 24, 2016 10:30 PM EDT Reads: 1,445
Aug. 24, 2016 09:45 PM EDT Reads: 4,782
Thomas Bitman of Gartner wrote a blog post last year about why OpenStack projects fail. In that article, he outlined three particular metrics which together cause 60% of OpenStack projects to fall short of expectations: Wrong people (31% of failures): a successful cloud needs commitment both from the operations team as well as from "anchor" tenants. Wrong processes (19% of failures): a successful cloud automates across silos in the software development lifecycle, not just within silos.
Aug. 24, 2016 08:15 PM EDT Reads: 1,992
There's a lot of things we do to improve the performance of web and mobile applications. We use caching. We use compression. We offload security (SSL and TLS) to a proxy with greater compute capacity. We apply image optimization and minification to content. We do all that because performance is king. Failure to perform can be, for many businesses, equivalent to an outage with increased abandonment rates and angry customers taking to the Internet to express their extreme displeasure.
Aug. 24, 2016 07:45 PM EDT Reads: 2,346
Right off the bat, Newman advises that we should "think of microservices as a specific approach for SOA in the same way that XP or Scrum are specific approaches for Agile Software development". These analogies are very interesting because my expectation was that microservices is a pattern. So I might infer that microservices is a set of process techniques as opposed to an architectural approach. Yet in the book, Newman clearly includes some elements of concept model and architecture as well as p...
Aug. 24, 2016 07:15 PM EDT Reads: 10,759
A company’s collection of online systems is like a delicate ecosystem – all components must integrate with and complement each other, and one single malfunction in any of them can bring the entire system to a screeching halt. That’s why, when monitoring and analyzing the health of your online systems, you need a broad arsenal of different tools for your different needs. In addition to a wide-angle lens that provides a snapshot of the overall health of your system, you must also have precise, ...
Aug. 24, 2016 06:30 PM EDT Reads: 1,494
SYS-CON Events announced today that Venafi, the Immune System for the Internet™ and the leading provider of Next Generation Trust Protection, will exhibit at @DevOpsSummit at 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Venafi is the Immune System for the Internet™ that protects the foundation of all cybersecurity – cryptographic keys and digital certificates – so they can’t be misused by bad guys in attacks...
Aug. 24, 2016 04:15 PM EDT Reads: 2,555
Monitoring of Docker environments is challenging. Why? Because each container typically runs a single process, has its own environment, utilizes virtual networks, or has various methods of managing storage. Traditional monitoring solutions take metrics from each server and applications they run. These servers and applications running on them are typically very static, with very long uptimes. Docker deployments are different: a set of containers may run many applications, all sharing the resource...
Aug. 24, 2016 03:00 PM EDT Reads: 1,868
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addres...
Aug. 24, 2016 02:00 PM EDT Reads: 1,838
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportuni...
Aug. 24, 2016 12:00 PM EDT Reads: 3,850
DevOps at Cloud Expo – being held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Am...
Aug. 24, 2016 11:45 AM EDT Reads: 3,360
Modern organizations face great challenges as they embrace innovation and integrate new tools and services. They begin to mature and move away from the complacency of maintaining traditional technologies and systems that only solve individual, siloed problems and work “well enough.” In order to build...
SYS-CON Events announced today that eCube Systems, a leading provider of middleware modernization, integration, and management solutions, will exhibit at @DevOpsSummit at 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. eCube Systems offers a family of middleware evolution products and services that maximize return on technology investment by leveraging existing technical equity to meet evolving business needs. ...
Aug. 24, 2016 11:00 AM EDT Reads: 421
DevOps at Cloud Expo, taking place Nov 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 19th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long dev...
Aug. 24, 2016 11:00 AM EDT Reads: 2,113