Your daily (lunar) base where you will find all the lightning-related news outer space.
Helping the network with your Lightning node
This week, in Episode 21, we talk with Carsten Otto, a developer and Lightning Network Node specialist from Germany. His routing node is at the top of the BOS score list!
He started in 2018 building his first node. And from May 2021 he really started building his routing node! He said that it was a combination of boredom and a rekindling of his initial excitement about the Lightning Network: “I wanted to understand what my node was doing, how it is being used, and how I could help the network by running it.”
What was your first experience with the lightning network and do you still know what you did/saw at that moment?
I read the Poon/Dryja paper and was confused. Then I read the BOLTs (specification), sitting on my balcony in Munich, enjoying the sun and some kind of drink. It took me a while, but the more I understood, the more I loved the idea.
If you look back on last year, what is the most stunning thing (thing/application/movement) you have seen on the lightning network?
Plebnet and RoFs/triangles. Lots and lots of people of various backgrounds and skill levels who managed to get their node up and well-connected. I see lots of chat groups, tutorials, videos, tools. I really enjoy people being curious about something, and it feels great to see them joining the network with maintained nodes. This is so much better than my early days, where I had no idea what I was doing. My node didn’t have any good channels, and I didn’t have access to a community that could help me get started.
We have LND (lightning labs), c-lightning (blockstream), eclair (Acinq) and some more… These are all implementations on the second layer. What lightning implementation do you prefer at the moment?
I only have experience with lnd. Even though there are some issues, I don’t see the need to invest a lot of effort into getting to know other implementations. I believe that interoperability is important, and I believe that other implementations made some great certain design choices. As long as there’s enough variety and good interoperability, I hope that you can’t really make a mistake picking one of the implementations at random.
Will the Lightning Network be mature enough for mass adoption within 4 years?
4 years isn’t that long for a decentralized system. There’s no “benevolent dictator” like Linus Torvalds. We have to agree on protocol changes, and we have to overcome the engineering challenges. Lots of money is at stake, and risking node operator’s money just to add a feature isn’t always a good idea. If you look at Christian Decker’s eltoo idea (proposed in 2018) and #pickhardtpayments you can see how slow development can be. Based on that I’d say no. Adoption will grow, and the LN will serve way more transactions than the Bitcoin blockchain could, but in 4 years time the LN won’t be ready for millions (or more) of users.
What is the most important thing to get fixed before we can say lightning succeeded and we reach mass adoption?
What is the most important thing to get fixed before we can say lightning succeeded and we reach mass adoption?
What’s the biggest problem you encountered when starting your first Lightning node?
It took me a long time to understand what’s going on. To me, understanding the outputs of lncli closedchannels / listchannels / pendingchannels and getting to know what kind useful information hides in the logs was very time consuming. One of the basic “things” in the LN, a payment that is forwarded along a few nodes, is extremely complex. It really helps to understand how HTLCs are constructed and how this relates to force-closes, but in order to get there, you also need to understand a lot of other things (and improve your tooling).
Security on lightning networks is at the moment one of the biggest topics at the moment. What are at the moment the biggest issues with security?
I didn’t know that this was a big topic. I think the issue of having a “hot wallet” is rather obvious, but I don’t have anything interesting to say about that.
There is a lot of discussion about the ranking of Lightning Nodes. Do you think it is important and why is that?
First of all, those rankins are only concerned with routing nodes. If you’re a merchant or service provider, you don’t really care about those rankings. I think that Terminal Web provides some good hints to node operators, i.e. that they need to have more reachable/routable channels, or better uptime. It doesn’t really matter if your node is ranked at #100 or #30, though. Making sure you’re ranked (and “good”) is the important step.
We talked with Severin Bühler from LnRouter.app about how important speed and deliverability is for the adoption of the Lightning Network. Nobody wants to wait 8 seconds for a payment to be processed. Do you agree with this and what do you think can help to make the Lightning Network to be much faster in the near future?
I think 8 seconds are fine for some forms of payment, and most forms of payment I’m used to in Germany take longer than 8 seconds. But yes, just hitting a button and getting the result in a second or two would be great. I’d think lots of work should be invested into the routing algorithms. Ideas like #pickhardtpayments aind BOLT 14 come to mind. Currently we only think about fees and add a bit of reliability/probability information to the mix, but it’s far from perfect. We should reason about latency, allow payments to be more expensive where speed is more important, reward nodes that help us with our needs instead of trying countless nodes without any reputation first.
We share the same goal for Lightning to reach critical mass so that more people can use Lightning for their daily living. What is the most important development we need to get more merchants to accept Lightning?
I think it takes a lot of effort to add something to existing solutions that already work. It’s a risk, and an investment. The process should be as easy as possible, and the integration should behave similar to the existing solutions. Automatic conversion to fiat might be very useful. Integrating with existing platforms might be necessary. I believe that incremental changes are easier, helping the LN economy to grow faster than adding “perfect” solutions to a smaller number of merchants.
Not so long ago we spoke with Hakuna as a guest in Episode 16. He’s an IT specialist from Germany and runs a very successful routing Node. He wrote the script “bash-rebalcer” as an extra feature to your “rebalance-lnd”.
He said and I quote: Carsten’s original script has the most clever algorithm to take some of the decisions away from you, but it requires you to be comfortable with command-line, and doesn’t allow batch running. I wanted to polish my bash scripting skills, and thought about a layer which allows for scale and simplicity. So the wizard allows beginners to get into rebalancing, but also allows professionals to move a few million around semi-automatically.
Do you know this script? And do you have plans of your own for some future episodes of “rebalance-lnd”?
I think I had a look at that script a while ago, but I don’t know enough about it. In general I’d like people to work on usability aspects. Adding a UI would be nice, or coming up with tutorials, or wrapper scripts. It’s open source, after all! Personally, I have the rough plan to add rebalancing features to lnd-manageJ, so that certain steps can be performed fully automatically (which is the “batching” Hakuna mentioned), taking into account the insights lnd-manageJ has about flows, peer/channel history, …
Do you like this article and our other content? Donate and support us in our mission to create a worldwide distributed Lightning Network by connecting countries around the world and bringing The Lightning News to you first!
Follow The Daily Moon also on Telegram and Twitter!
lnd-manageJ is intended to run 24/7 in the background to collect statistics about your node.
It helps you understand and manage the inner workings of you lnd node.
What usefull information can you collect with running this ?
Some of the cooler features: How much did I already pay for force-closes (including sweep transactions) for some peer/channel? How many sats did some node/channel send/receive in the past X days, and how much of that is due to rebalancing? How much did a node/channel contribute as a rebalance source, i.e. how many sats did it provide which I put into some other channel that I wanted to “fill” by running rebalances?
In general lnd-manageJ helps me get all kinds of information that might be hidden or hard to compute if you only have access to the output of lncli and your log files.
You started in 2018 with your first node and in 2020 you decided to build/expand a routing node.
What was the trigger to this?
I think I only started taking care of my node in May 2021, I don’t know about anything interesting (node related) that happened in 2020, possibly aside from moving lnd to a different server and picking the alias “c-otto.de”. I don’t remember what exactly made me take care of my node, but I it was a combination of boredom and a rekindling of my initial excitement about the LN. I wanted to understand what my node was doing, how it is being used, and how I could help the network by running it.
Some of our previous guests said that it’s most likely that in the future smaller nodes will disappear and bigger routing nodes with high liquidity channels will remain. What’s your vision on it ?
Bitcoin and LN are permissionless AND extremely interesting. Because of that I believe there will always be several smaller nodes, run by hobbyists and those who want to do their part against centralization. However, I also think that having more liquidity and having “professional” hardware/monitoring/… is necessary to run a successful node that’s used by many people.
For those who want to run a routing node the most valuable asset is to know where the traffic is.
How do you find this information?
First, you can experiment yourself. If you have good inbound liquidity, you can see which of your outbound channels are used, and which are idle. You can tweak your fees to get more insights. You can open new channels and see if they attract traffic. Other than that, lnrouter.app has some good suggestions, and you can also look at the fee rates of nodes like mine to see which nodes require lots of inbound liquidity.
What mistakes do you often see new node owners make?
I believe many node operators are obsessed with balanced channels, in the sense of 50/50. As the author of rebalance-lnd, which for several years tried to reach this 50/50 target, I know the feeling. However, nowadays I believe that it’s more important to have enough liquidity in channels that are large enough.
On your site you describe channel and fee Policy from your node and some other cool things.
Are 3M sats channels the minimum size if you want to be a routing node?
I think we need more categories for routing nodes. One can be a liquidity battery, where several Bitcoin worth of liquidity are made available so that this node helps you reach your destinations reliably. These routing nodes need larger channels, and I think 10M+ should be the lower bound for “interesting” peers like exchanges or service providers. The other category is “edge routers”. Those need to connect to “dark” corners of the network, or fill gaps between two parts of the network. You don’t need a lot of liquidity for these types of channels, but I believe that on-chain costs make smaller channels less attractive.
Another tool that a lot of members of our community use is charge-lnd. With this script Node Operators can automate setting their fees. Can you explain how rebalance-lnd takes into account the future profit you can make on a channel instead of only using proportional rebalancing with charge-lnd?
Rebalance-lnd only considers the fee rates at the time of starting the script. There’s no history of old fees, no knowledge of “magic” like what charge-lnd does, and the script also doesn’t guess how the future might look like. It’s rather simple, in that sense. The script just assumes that the amount you move around as part of the rebalance transaction is routed to the other side, at the current fee rate. If you move 300k sat to a channel where you charge 2000ppm, the script just assumes you’ll earn 600sat in the near future. It also considers the fees you have to pay for the rebalance transaction, let’s say for example 100sat. In total the profit could be 500sat. If you’d earn more than 500sat by routing 300k sat from the original channel, it would be a bad idea to do the rebalance transaction, and rebalance-lnd wouldn’t let you do it. All of this depends on the fee rate, and everything changes if you charge more or less on those two channels. I don’t want to criticize proportional fees too much because many people seem to be happy with it. To me, it seems wrong to lower fees on a channel just because you have a lot of liquidity. Maybe the high fee rate is fine, and you just need to wait a day or two for some large transaction to come through? Why do you want to raise fees only if you don’t have a lot, why not use the higher fee rate for other amounts, too? It’s tricky.
Some members of our community use Zero Base Fee. Of course to support the algorithm and the network, but some also hope for more routing on their nodes. Do you think setting up Zero Base Fee to get more routing?
Today? No. Tomorrow? No. In a month? Maybe. In a year? Yes! #pickhardtpayments will only pick channels with 0 base fee (and if there is a way to allow channels with a non-0 base fee, those will not be as attractive). As such, if #pickhardtpayments are used, node operators with 0 base fee with see more traffic.
You build the rebalance-lnd script to help you rebalance your channels. It’s a real must have for a lot of Node Operators. Why did you build it and what is the best tip for users who start to use it?
When I started running my node, I heard about the importance of having balanced channels, without really understanding the details of that. However, I wanted to fix the issue for myself and started coding the script, because I didn’t find any other tool or command that could help me rebalance my channels. The current version is very different from the one I started with. Nowadays I suggest node admins to just run the script when they want to have more outbound liquidity in a specific channel, assuming they don’t want to open a new channel to that peer. Just target the channel, make sure your fees are realistic, pick some amount around 100k sat, and go. That’s it. Do that a few times until you’re happy with the results, and maybe add some scripts to automate things if you feel the need.
We saw examples of Node Operators using charge-lnd and rebalance-lnd combined? What do you think of that solution? And what is the most important for users to be aware of when automating?
I don’t think it’s a good idea, and it might be risky. One simple issue can be that you use rebalance-lnd to “fill up” a channel, so that charge-lnd lowers the fees. This way you’d earn less in the future, working against the logic that rebalance-lnd used just a minute ago. Whenever you automate something, you need to work in all the decisions that you usually make manually. All safety checks. Make sure the script doesn’t run too often. Make sure it doesn’t do more than you want it to do, for example “rebalancing” a channel until 100% is on your side. Don’t let a script change your fees unless you’d also change the fees manually.
We heard Rene Pickhardt talk about that larger channels will be preferable in path finding for payments and thus getting Node Operators more routing. Is this true? And what channel liquidity would you recommend to Node Operators that want more Routing on their node?
Yes, the core result of #pickhardtpayments is that larger channels should be preferred as they are the best choice given the limited information one has about the network. However, a large channel is only helpful if it connects the sender to the recipient, or at least helps bringing the payment closer to the recipient. As such, one one side node admins should open just one massive channel to the destination, to optimize the chances of being picked for forwarding requests. On the other hand there’s not just one destination, so instead of opening one massive channel it might be better to open two channels each half of that size. If you operate a node with a few BTC or more, I believe that channels should be at least 10M sat in size, at least for channels to “big players” like exchanges.
Will multi part payments cause node operators to open less huge (WUMBO) channels in the future ?
If #pickhardtpayments are used, I believe people will try to have larger channels, not the other way around. In simple terms, if the algorithm has to pick between a 10M sat channel and a 50M sat channel, the larger channel is far more attractive (due to the probability that it has enough for the amount I try to send). As such, node operators with larger channels may attract more routing transactions, and they may charge higher fees.
One of you repositories on github is Bitbook.
The purpose of BitBook is to track your own coins by organizing on-chain transactions and their corresponding addresses. How did you come with this idea? What ‘s the purpose of it ?
I got Bitcoin from various sources, including some faucets that gave away small amounts for free. Over time I used several wallet implementations, used hundreds of addresses. It felt like I lost track of the details. Instead of relying on my messy notes, I wanted something “real” that show the movement of my coins with more confidence. In addition to that, I always believed that I might have some forked coins which I could sell. For this, I needed to know exactly how many Bitcoin I had at which address at the time of these forks.
You did a fork of the multi part payments repository from Rene Pickhardt & Stefan Richter (MPP splitter)
What did you optimize with your work ?
I only forked the repository to fix some typos 🙂 So far, I haven’t used #pickhardtpayments for anything useful, but I’m adding lots of code to lnd-manageJ that hopefully can be used soon. For me, a good next step would be to see more and more people actually experimenting with their own nodes, so that we can gather more data and see the benefits of #pickhardtpayments. I believe that having a solution that works with lnd would open a lot of doors, possibly even helping node operators and businesses to send large MPPs without having support for #pickhardtpayments in lnd itself.
What kind of layer 3 solutions would you like to see soon?
I don’t have enough imagination for that. I’d prefer having a layer 2 that actually works, so that it’s extremely easy to add layer 3 ideas. Right now, I believe we still have a lot to do to make the LN attractive to more people and use cases.
Germany is one of the countries with the most Lightning nodes. What do you notice about the Bitcoin and Lightning adoption in the neighborhood where you live, Cologne? Any merchants nearby?
Not that I’m aware of. I’m mostly stuck indoors, though, due to Covid19.
Rene Pickardt & Stefan Richter observed that finding the cheapest multi-part payments is an NP-hard problem considering the current fee structure and propose dropping the base fee to make it a linear min-cost flow problem. Will the base fee disappear in the future maybe with BOLT 14?
BOLT 14 is designed to improve the information that nodes have about the network. When sending a payment, instead of failing again and again, pathfinding algorithms could make use of that information and improve the user experience by finding a good route faster. As such, both BOLT 14 and #pickhardtpayments are designed to make payments in the LN more reliable. However, I don’t think BOLT 14 is related to (zero) base fees, at all. The algorithm behind #pickhardtpayments does not work with non-zero base fees, and channels that have a base fee configured are simply ignored. Because of this, once enough payments are sent using #pickhardtpayments, node operators are incentivized to drop the base fee to zero, so that they can benefit from these transactions. I believe more node operators will drop the base fee once #pickhardtpayments become a reality. Keeping the base fee won’t make you rich, and you won’t stop other nodes from probing your node for free.
Do you see a future in layer 2 solutions for Bitcoin like RSK which bring the functionality of Ethereum Smart Contracts to the Bitcoin blockchain?
I have no idea what RSK is and I didn’t even bother googling it. Although I’ve heard the term “Smart Contracts” before I knew what Ethereum was, I still don’t see the point. As long as I don’t see valid use cases for smart contracts, I’m not interested in technical solutions involving smart contracts.