2019-09-12

Ballerina Programming Language - Part 1: Concept

In the previous post, I talked about the context for Ballerina. In this post, I want to explain what kind of programming language it is. We can summarize this as a number of design goals:

  1. Provide abstractions for networking
  2. Use sequence diagrams as the visual model
  3. Minimize cognitive load
  4. Leverage familiarity
  5. Enable a semantically-rich static program model
  6. Provide a complete platform, not just a language
  7. Allow multiple implementations, based on different runtime environments

This is not exactly the list we started out with: it has evolved in the light of experience.

I will talk about each of these points in turn. The first two of these are a bit different. They correspond to the two fundamental features that make Ballerina unique.

Networking

The primary function of an ESB is to send and receive network messages. So language-level support for this is central to the Ballerina project.

On the sending side, the key abstraction is a remote method. A remote method is part of a client object; there is a distinct syntax for calling remote methods. A program sends a message by calling a remote method; the return value of the remote method can describe the response to that message. Implementation of remote methods is typically provided by library code or is auto-generated; each protocol will have its own client object implementation. Application code calls remote methods on client objects..

On the receiving side, the key abstraction is a resource method; a resource method is part of a service. Application code provides services by implementing resource methods. This works in conjunction with listener objects. Implementation of listener objects is typically provided by library code; each protocol will have its own listener object implementation. Listener objects call resource methods on services provided by application code.

There's a final twist that ties together the sending and receiving side: resource methods are typically passed a client object to allow them to send messages back to the client.

So what is gained by providing language-level support for networking?

Most importantly, it enables a visual model that shows the program's behaviour in terms of these abstractions: the visual model can show how the program interacts using network messages. This links up with the second unique feature of Ballerina - the use of sequence diagrams as the visual model.

It also provides a purpose-designed syntax, which does not require the developer to jump through a series of hoops. You use the language-provided syntax and it just works. It's as easy as writing a function. This is not all that important for a large program. But many programs that perform integration tasks are small, and with small programs reducing the ceremony matters. You could compare this with how AWK takes care of opening files and iterating over each line of the file: it's not hard to do, but for a small program the fact that AWK takes care of this for you is a significant convenience.

Related to this is that Ballerina's model of program execution incorporates the concept of running as a service. You don't have to write an explicit loop waiting for network requests until you get a signal. The language runtime deals with all that for you. Again, not revolutionary, but it makes a difference.

The final advantage relates to typing. At the moment the type of a resource method is just a function type, and the type of a service is just a collection of these function types. But we want to do better than this. The type of a resource method should capture not just the type of its parameters and return value, but the type of the messages that it expects to receive, and the type of the message that it will send in response. (The former is at the moment partially captured by annotations, which can be generated from, for example, Swagger/OpenAPI descriptions) Furthermore, the type of a service should capture not just the type of each message exchange separately, but also the relationship between the exchanges. This is usually called session typing and is an active area of research.

Sequence diagrams

WSO2's experience from working with customers over many years has been that drawing a sequence diagram is typically the best way to describe visually how services interact. An ESB's visual model is typically based on dataflow model, which works well for simple cases but is not as expressive. So one big idea underlying Ballerina is that you should be able to visualize a function or program as a sequence diagram.

It is important to understand that the visualization of Ballerina code as a sequence diagram is not simply a matter of tooling that is layered on top of the Ballerina language. It took me a long time to really grok Sanjiva's concept for how the language relates to sequence diagrams. My initial reaction was that it seemed to me like a category error. Sequence diagrams are just a kind of picture. What's that got to do with the syntax and semantics of a programming language?

The concept is to design the syntax and semantics of the language's abstractions for sending network messages, for in-process message passing and for concurrency so that they have a close correspondence to sequence diagrams. This enables a bidirectional mapping between the textual representation of a function in Ballerina syntax and the visual representation of the function as a sequence diagram. The sequence diagram representation fully shows the behaviour of the function as it relates to concurrency and network interaction.

The closest analogy I can think of is Visual Basic. The visual model of a UI as a form is integrated with the language semantic to make writing a Windows GUI application much easier than before. Ballerina is trying to do something similar but for a different domain. You could think of it as Visual Basic for the Cloud.

Cognitive load

Programming languages differ in the demands they make of a programmer. One way to look at this is in terms of different developer personas, such Microsoft's Einstein, Elvis and Mort personas. But it's hard to do that without implying that one kind of developer is inherently superior to another, and I don't think that's a helpful way to look at things. I prefer to think of it like this: a programming language both gives and takes. It gives abstractions to make it convenient to express solutions, and it gives the ability to detect classes of errors at compile time. But it takes intellectual effort to understand the abstractions that are provided and to fit the solution into those abstractions. In other words, it relieves the programmer of some of the cognitive load required to write and maintain a program, but it also imposes its own cognitive load. Every programming language needs to strike a balance between what it gives and what it takes that is appropropriate for the kind of program for which it is intended to be suitable.

For Ballerina, the goal has been for it to make only modest demands of the programmer. Integration tasks are often quite mundane; people just want to get things working and move on. But these integrations, although mundane, can be critically important to a business: so they need to be reliable and they need to be maintainable. So the language tries to nudge programmers in the direction of doing things in a reliable and maintainable way.

Familiarity

One way to reduce cognitive load is to take advantage of people's familiarity with programming languages. Specifically, Ballerina tries to take advantage of familiarity with programming languages in the C family, such as C, C++, Java, JavaScript and C#. This applies to both syntax and semantics. It is not a hard and fast rule, but a guideline: don't be different from C without a good reason, and elegance does not by itself count as a good reason. A good example would be the rules for operator precedence: the C rules are quite a bit different from what I would design if I was starting from scratch, but the benefits from better rules just aren't enough to make it worth being different from all the other languages in the C family.

Semantically-rich static program model

I have struggled to find the right phrase to describe this. It is a generalization of static typing. The idea is that the language should enable programs to describe their semantic properties in a machine-readable way. The objective is to enable tools to construct a model of the program that incorporates these properties, and then use that model to help the developer write correct programs. This ties up with the cognitive load point. A semantically-rich model enables more powerful tools, which help reduce the effective cognitive load on the programmer

"Static" means that a tool can build a model of the program just by analysing the source code, without needing to execute the program. Often this is called "compile-time", but that doesn't seem appropriate for the way an IDE will use this model. Visual Basic used to call it "design time", but that seems a bit narrow too: continued maintenance is just as important as initial design.

For types, it means we want a static type system. But our approach to static typing is pragmatic. The static type system is there to help the programmer. We don't want the static system to be so sophisticated or so inflexible that it becomes an obstacle to writing programs. The goal is not to statically type as much as possible, but to statically type to the extent that it is likely to be helpful to the programmer writing the kinds of program for which we intend Ballerina to be used.

Types are just one kind of semantic richness. There are many others.

  • Sequence diagrams depend on building a model of the program where sends and receives are matched up.
  • Documentation that is structured, not simply free-form comments, can be checked for consistency with the program, and can be made available through the IDE.
  • Properties of services and listeners can be used to automate deployment of Ballerina programs to the cloud.

Platform

There's a distinction between the core of a programming language, which defines the syntax of the language and the semantics of that syntax, and the surrounding ecosystem. Often the core comes first, and the ecosystem develops organically as the core gains popularity. A lot of the utility of any language comes from the surrounding ecosystem.

In Ballerina, we refer to the core as "the language" - it's the part that's defined in the language specification. With Ballerina, the language has been designed in conjunction with key components of the surrounding ecosystem, which we call the "platform".

The platform includes:

  • a standard library
  • a centralized module repository, and the tooling needed to support that
  • a documentation system (based on Markdown)
  • a testing framework
  • extensions/plug-ins for popular IDEs (notably Visual Studio Code).

This all takes a lot of work, and is a big factor in why Ballerina has required such a large investment of resources from WSO2.

Multiple implementations

Although the current implementation of Ballerina compiles to JVM byte codes, Ballerina is emphatically not a JVM language. We are planning to do an implementation that compiles directly to native code and we've started to look at using LLVM for this. I suspect that an implementation targeting WebAssembly will also be important long-term.

We have been careful to ensure that the language semantics, particularly as regards concurrency, are not tied to the JVM. This was part of the motivation for the initial proof-of-concept implementation approach, which compiled into bytecode for its own virtual machine (BVM), which was then interpreted by a runtime written in Java. Although the 1.0 implementation compiles directly to the JVM, it is not an entirely straightforward mapping; it takes some tricks to implement the Ballerina concurrency semantics (similar to how Kotlin implements coroutines).

There are languages that are defined by an implementation and there are languages defined by a specification. For a language with multiple implementations, it is much better if the language is defined by a specification, rather than by the idiosyncrasies of a particular implementation. The Ballerina language is defined by its specification. This specification does not in any way depend on Java.

Initially, the specification was a partial description of the implementation. But now we have evolved to a situation where the implementation is done based on the specification. From a language design point of view, we are ready for multiple implementations. It is "just" a matter of finding resources to do the implementation. One of my hopes in writing this sequence of blog posts is somebody outside WSO2 will feel inspired to their own implementation. We would be more than happy to work with anybody who wants to take this on.

Conclusion

There has long been a distinction, originally due to John Ousterhout, between systems programming languages, and scripting or glue languages. Systems programming languages are statically typed, high performance and designed for programming in the large. Scripting/glue languages are dynamically typed, low performance and designed for programming in the small.

There are elements of truth in this distinction, but I see it more as a spectrum than as a dichotomy, and I see Ballerina as being somewhere in the middle of that spectrum. It has static typing, but it's much less rigid than the kind of static typing that systems programming languages have. It is capable of decent performance: it should be possible to make it quite a bit faster than Python, but it will never rival Rust or C++. It's not designed for programs with hundreds of thousands of lines of code, but it's also not designed for one-liners. Here's how I would place Ballerina on the spectrum relative to some other languages:

  • Assembly
  • Rust, C
  • C++
  • Go, Java, C#
  • Ballerina
  • TypeScript
  • Python, JavaScript
  • PowerShell, Bourne shell, TCL, AWK

Go is a bit hard to place relative to Java/C#. In some ways, it's more on the systems side (no VM); in some ways, it's more on the scripting side (typing). I would put Ballerina between Go and TypeScript.

In future posts, I will get into the concrete language features that these design goals have led us to. The details of the core language are in the language specification. The rest of the platform does not yet have proper specifications, but there is lots of documentation on the web site.

6 comments:

Hans-Jürgen Rennau said...

Ballerina is about "enterprise integration", thus also data integration. In this context, navigation (within and across structured values) and transformation (of structured values) are key operations - at least on the conceptual level. (Their mappings to programming language constructs are a different matter, unfortunately.) I wonder if the language design of Ballerina reflects a desire to simplify navigation and transformation?

James Clark said...

That's a great question. I should have said something about that in my blog post. The answer is emphatically yes. The whole design of the language is what you might call data-centric (as opposed to object-oriented).

There is a lot of functionality related to this in the works:

- stabilizing the table datatype
- adding language-integrated query, which works with both tables and other datatypes
- stabilizing the XML datatype
- adding XML navigation (similar to XPath)

There are proposals for the above.

We also want support for "data-in-motion". There is an experimental streams feature that does this.

Probably further off, we want to have a more declarative way of doing transformations, which would support a graphical interface, based on dataflow, similar to ESBs (this is issue 39).

Hans-Jürgen Rennau said...

Thank you very much for this clarification and the valuable links, food for thought and study!

There is another aspect I feel deeply curious about: the concepts of resource and resource identifier. Roy Fielding defined: a resource is a mapping of a resource identifier (URI) to a set of representations, an abstraction at the heart of the REST architectural style and the engine of generic link traversal. But the Ballerina point of view seems to be different: a resource is an endpoint attached to a set of methods, so resource is a synonym for a deployed service. Is this right? Or is the Ballerina perspective perhaps dual, layering the service perspective – helpful for orchestrating tightly coupled services – on top of Fielding’s perspective, enabling link traversal, REST services and boundless connectivity?

James Clark said...

Ballerina's network abstractions are at a different level of abstraction/generality to REST. They are intended to work with a wide variety of protocols, such as WebSockets or AMQP, not just as HTTP. Most of them are on top of IP but that's not inherent: you could use them to talk to a VT100 terminal over RS232 if you wanted to. You can implement distributed applications in the REST architectural style on top of Ballerina, but there's nothing specific to REST in Ballerina. A resource is just the entry point from the network to user code. A service ties together resources that are related, e.g. because they correspond to different messages that are part a single protocol.

Sanjiva Weerawarana said...

While I agree with James' answer, in the case of HTTP, the resource identifier is / by default. Doing a GET on the the resource ID invokes the resource function which offers a representation of the resource based on content negotiation.

In other words, its possible to write fully restful (including HATEOS) services in Ballerina very cleanly using the service/resource abstraction.

Hans-Jürgen Rennau said...

Thank you very much, both of you! If I am not mistaken, our uses of the term REST are quite different. Yours is the widespread one, denoting a style of web service design, including an HTTP-based elaboration. Mine is the original one as defined by Fielding in his dissertation: REST as an architectural style = set of constraints applied to the elements within the architecture. This set of constraints has been defined for the sake of their joint impact on key properties of network-based applications (performance, scalability, simplicity, modifiability, visibility, portability, reliability). So I wonder if it made sense to define an own architectural style, reflected by the specification and the implementation of Ballerina. Such a definition might be very valuable as giving guidance for future development (and identifying issues in the current state). That is exactly the role REST played in the development of the internet standards.