Routing traffic between netifs #537
Comments
|
Do you want to route (L3; i.e. on IP level?; still might be possible, will try to dig it out) or is L2 bridging also good? For the latter case, see this issue: #508 |
|
Ah unfortunately bridging seems to work only when the wifi is in AP mode though, probably not your use-case... |
Ideally on L3? But really I don't care so long as the receiving netif knows to automatically take L2 packets and forward them to its IP layer? I'm not sure about that. I'll look at everything you sent. On this issue and the other |
|
What I need is to be able to route/bridge my wireguard netif that's been created in the C layer with the ethernet "router" netif I created in rust. Ideally this would be done in rust but I'm starting to believe I'm going to have to make my own wrappers here. I would just like to know what functions I should be using for this, or if it's a lost cause, just a nudge really. |
let mut eth_netif = EspEth::wrap_all(
eth_driver,
EspNetif::new_with_conf(&NetifConfiguration {
flags: esp_netif_flags_ESP_NETIF_DHCP_SERVER | esp_netif_flags_ESP_NETIF_FLAG_AUTOUP,
got_ip_event_id: NonZeroU32::new(ip_event_t_IP_EVENT_ETH_GOT_IP as _),
lost_ip_event_id: NonZeroU32::new(ip_event_t_IP_EVENT_ETH_LOST_IP as _),
key: "ETH_DEF".try_into().unwrap(),
description: "eth".try_into().unwrap(),
route_priority: 50, // Higher is better
ip_configuration: Some(IpConfiguration::Client(IpClientConfiguration::Fixed(IpClientSettings {
ip: Ipv4Addr::new(10, 10, 10, 1),
subnet: Subnet {
gateway: Ipv4Addr::new(10, 10, 10, 1), // This is the gateway advertised by the dhcp server
// real_gateway: IpV4Addr::new(69, 69, 69, 69) // This would be the gateway this if routes its packets to
mask: Mask(30),
},
dns: None,
secondary_dns: None,
}))),
stack: NetifStack::Eth,
custom_mac: None,
})?,
)?;To recenter the problem, I need a way to differentiate the gateway that the dhcp server advertises to the network, and the gateway that the netif itself will use to connect to the internet. Here, that would be the ip of my wireguard netif. Otherwise I'm a bit stuck. |
OK now clear. But then your use case is definitely then routing and not bridging, in that you are creating a mini router (+ quite possibly, NAT). Also, I think for a proper router you absolutely need two netifs, and not one:
Now, honestly I'm not sure how to do the routing & NAT excercise. But there is something in ESP-IDF called "NAPT" which is badly documented, but used to work back in time in a now-archived old project of mine. I think this was a simplistic "NAT+router", but you need to google a bit on the Internet as to exactly what it was as I am actually having a hard time recollecting my memories. For one, I'm not sure why I don't have any code that connects two netifs together (the LAN and WAN one) when enabling NAPT. It could well be, that the two netifs in question are the Wifi STA and AP netifs, hence no need for any explicitness. But you probably want instead eth as LAN and Wifi STA as WAN (or another eth as WAN). |
|
Thanks i'll look all this up. I had another idea elsewise, though I don't know what's it's worth. What if i create the eth netif with gateway=ip as I did before, advertise that to the pc, then sneakily replace the netif with another that's got the gateway of what I need? So long as we don't say anything, the pc should be none the wiser, no? And have the right gateway to boot. The only issue I can think of is when the lease expires we're kinda screwed. And lwip only allows the lease to go up to an hour. |
|
The other idea was to go to L2 and try to bridge the two netifs there to bypass the whole DHCP+routing problem but I don't know how feasible that is. |
Bridging means a single ip for all netifs in the bridge which is not what you want? I don't think either this or the earlier hack would work. After all, you want to correctly pass packets between two different networks so you have to properly do routing. And moreover - likely nat too. |
I'm being stupid with my language, what I meant by "bridge" here is "make callbacks between the two netifs so that they forward packets to each other" |
Sure BUT:
Above might be possible but I wonder why not just giving NAPT a try? |
Oh I certainly will, I was just throwing ideas in the wind first. |
|
But wait so we have a wrapper to enable napt but not to do anything with it? Or am I blind? |
|
Well that clearly wasn't it. I'm kinda lost. I did this: log::info!("Enabling napt on eth netif..");
// Necessary for routing packets between subnets.
eth_netif.netif_mut().enable_napt(true);Now what? Nevermind wrappers, i don't even understand what C functions I have to use now that I enabled napt. |
You don't have to use any extra C functions. Just make sure you have another netif (wifi or eth) which is configured to your 67.67.67.67 network and the routing should just start happening automatically between the two the moment you call |
|
I can't clone the eth driver i'm wrapping to make the new ethernet netif. |
I do not understand what you are trying to achieve with that. Of course you can't and should not clone. And of course you can't and should not have multiple netifs assig ed to the same eth driver. If you explain what setup you are actually trying to achieve (say, with terminology as if you were using a regular pc networking) I might be able to help /suggest something, but otherwise I cannot explain to myself why you are not doing two drivers / netifs as I already suggested. As in your rmii eth + wifi sta. Or your rmii eth + an spi eth. Unless you use vlans, you anyway need two physical mediums for routing. |
|
Ok so what I have, is a setup like this: What I need is to be able to send packets from eth to wg0 and back. I don't really care how so long as they can speak to each other. |
|
I will know at runtime what the ip and subnet of the wgnetif is so it being variable should not be a problem, i just dont really know how to forward packets between the two netifs |
|
What is WgNetif in the PC world please? Tun? Tap? Something else? |
|
Wireguard virtual interface, so TUN. |
|
Then you might need something like this on the esp32 too. But what I don't get it is ... do you plan to run the wireguard vpn layer on the esp32 itself? Isn't it a bit underpowered for that? As in like 300kb ram (not counting psram) and relatively slowish? |
|
Hm, apparently it is possible, not sure how fast it is though: https://github.com/ciniml/WireGuard-ESP32-Arduino |
|
Separate from how the whole wireguard networking works (I need to read on that) you might have to tap into mbedtls or else it would be painfully slow. The mbedtls impl for esp idf have some if the algorithms implemented using hardware, so much faster (or to put it another way, not unbearably slow). |
|
But I think you are at a point where you need to explain your idea in more detail, as I'm at a loss as to what device you are trying to build. Like, is this some sort of wireguard vpn "device" and if yes, what is the point? |
|
The idea is to make a plug and play usb key that encapsulates both connectivity and vpn as transparently as possible for the user. Nothing must run on the computer. I've managed to get the tunnel to work but I don't understand how to make the wg netif speak to the eth netif so that the computer itself can speak to the internet through the vpn. |
|
Hmmm
Actually... I take my words back. let eth_handle: *mut esp_netif_t = eth_netif.lock().unwrap().netif_mut().handle();
let wg_handle: *mut netif = (*WG_CTX.lock().unwrap().0).netif;
esp!(esp_netif_bridge_add_port(bridge_handle, eth_handle))?;
esp!(esp_netif_bridge_add_port(bridge_handle, wg_handle))?; //Not workingIt is not working because |
|
T.b.h. I'm not really sure where this "67.67.67.67" network you were mentioning earlier even fits into the picture?
That's it? |
|
Btw you can do the above purely with Wifi, as the Esp can operate simultaneously in an AP+STA mode. No need for Eth or anything like that. |
|
This is a very heavyweight way to avoid converting Line 970 in 2968fa1 You just need to create a regular This means you need to do adaptation of the LwIP Wireguard code, but at least this is solving the "netif"-to-"esp_netif_t" problem? |
Yeah that was merely an exemple i used to illustrate the problem, it has no basis in reality.
Should be what I want. I think. The http server on the side would still see the request anyway right? The request would just be duplicated and also sent to the bridge which would then drop it or do whatever, which we don't care about. (I think.)
I need eth because i'm building an usb key, I have a schematic that converts usb to eth frames that are interpreted by the esp32.
I'll look into it, thanks. My main gripe with the whole conversion thing was I don't want to have to go through the thousands of lines of wireguard C code I have in the lib to modify everything to be esp_netif_t. If there's a way to wrap it in the rust layer, i'm all for it, otherwise i'll bite the bullet. |
Clear now.
That's also exactly how I understand it (or rather - hope it would work).
Fair enough. And really a side topic. Risking the derail the conversation even further: there is this s3 chip which I think was able to act as a real USB host. Where I'm going with that is that maybe you can even do it without any schematics, by implementing the USB-to-Ethernet purely in software with that chip. But yes, side topic.
Got it. But then I really don't know how easy it would be to wrap a Speculating: the other thing is, if you plan to have this device for commercial usage, it might be that this wireguard C code would anyway has to be changed or even completely rewritten (and then better in Rust I guess) if it is not fast enough. |
Do you still have the link for that? I'm very interested.
Yeah, that sounds like pain. Part of the reason I dread the first option not working.
Thanks I'll look at it. I'm a student though, this is all just a torture project thought up by my professor to rate us as either "passable" or blights upon humanity. So this isn't gonna be commercialized anytime soon if ever. And optimizing data flow rates is something I'll look at if I manage to make everything work seamlessly. Because yeah, iirc, the lib I use for wireguard writes its own implementation of the chacha/poylwhatever crypto functions. So maybe hooking everything to mbedtls instead wouldn't be the worst idea. An idea for later. |
Example with ethernet over usb: https://github.com/espressif/esp-idf/tree/master/examples/peripherals/usb/device/tusb_ncm
Fair enough! |
|
Ok so this whole thing won't work. Because apparently this works purely on an L2 level (should've known, what with the name and all..) and mac stuff gets beheaded at the subnet boundary. So back to figuring out how the hell nat works.. |
If I can't get the "easy" nat implementation to work I'll give this a try. |
|
Can you remind me how and in what context napt is supposed to be used here? You said I just had to enable it and then the netif would magically speak to each other but there has to be some assumed restrictions to this yes? It seems quite magical that they would do this without specific routes being set too.. And isn't napt supposed to be to communicate with the outside? why are we even using this to speak between subnets? I'm so deep into it I don't even remmeber why I started this |
|
The problem was I need a way to move traffic from my 10.10.10.0/30 subnet that contains the PC and the esp's ETH netif to the let's say 50.50.50.0/30 subnet (completely random made up number we don't care) that contains my WG netif and the other guy's netif.
|
That's what I thought initially and that's why I was constantly throwing at you the fact that such a bridge is an L2 thing, but I no longer think a bridge will NOT work. I now think it WILL work. Imagine that you have:
Then you bridge these two ^^^ and assign to the bridge netif to be a Router with IP subnet 10.10.10.0/30 and have the 10.10.10.1 gateway. What do you achieve this way? A lot, in my opinion:
|
|
So your idea is something like eth <-> bridge <-> wg netif but I don't understand why the bridge has the ip/subnet it does? Currently the idea is that my ethernet netif can serve an ip to the pc, that won't be possible with this design will it? And so when we say bridge here we don't actually mean the weird IEEE L2 bridge provided by esp idf do we? It's just the term we use for "random netif we hijack to do our bidding" yeah? |
When you put multiple network interfaces in a bridge, they are assigned the same ip address and in fact all netifs in the bridge do get even the same mac. From the pov of the other network peers the bridge is a single entity. That's why bridges are l2.
That's exactly what I hope you can use. What is so weird about it? |
|
Ok leaving that aside for a second, if I am to do any of the above, I need to be able to wrap my (*mut netif) into a (*mut esp_netif_t), and, ideally do better than that, which is to create the netif in rust directly using EspNetif::new_with_conf. struct esp_netif_obj {
// default interface addresses
uint8_t mac[NETIF_MAX_HWADDR_LEN];
esp_netif_ip_info_t* ip_info;
esp_netif_ip_info_t* ip_info_old;
// lwip netif related
struct netif *lwip_netif;
err_t (*lwip_init_fn)(struct netif*);
esp_netif_recv_ret_t (*lwip_input_fn)(void *input_netif_handle, void *buffer, size_t len, void *eb);
void * netif_handle; // netif impl context (either vanilla lwip-netif or ppp_pcb)
netif_related_data_t *related_data; // holds additional data for specific netifs
#if ESP_DHCPS
dhcps_t *dhcps;
#endif
// io driver related
void* driver_handle;
esp_err_t (*driver_transmit)(void *h, void *buffer, size_t len);
esp_err_t (*driver_transmit_wrap)(void *h, void *buffer, size_t len, void *pbuf);
void (*driver_free_rx_buffer)(void *h, void* buffer);
// dhcp related
esp_netif_dhcp_status_t dhcpc_status;
esp_netif_dhcp_status_t dhcps_status;
bool timer_running;
// event translation
ip_event_t get_ip_event;
ip_event_t lost_ip_event;
// misc flags, types, keys, priority
esp_netif_flags_t flags;
char * hostname;
char * if_key;
char * if_desc;
int route_prio;
#if CONFIG_ESP_NETIF_BRIDGE_EN
// bridge configuration
uint16_t max_fdb_dyn_entries;
uint16_t max_fdb_sta_entries;
uint8_t max_ports;
#endif // CONFIG_ESP_NETIF_BRIDGE_EN
// mldv6 timer
bool mldv6_report_timer_started;
#ifdef CONFIG_ESP_NETIF_SET_DNS_PER_DEFAULT_NETIF
ip_addr_t dns[DNS_MAX_SERVERS];
#endif
};This is in components/esp_netif/lwip/esp_netif_lwip_internal.h; except what we see in the bindings.rs file is this: #[doc = " @brief Type of esp_netif_object server"]
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct esp_netif_obj {
_unused: [u8; 0],
}
pub type esp_netif_t = esp_netif_obj;That's it. And it gets it from components/esp_netif/include/esp_netif_types.h: /** @brief Type of esp_netif_object server */
struct esp_netif_obj;
typedef struct esp_netif_obj esp_netif_t;Is it just not importing any include files that aren't in the include folder of a component? In which case we have a big big problem because this is just not doable. Or maybe I'm stupid. You tell me. I hope I'm stupid, this is painful enough as it is. |
|
Ideally, I'd do something like |
|
Welp, from what I've seen there's exactly 0 ways to do this except by extending the bindings ourselves, which is stupid, so I guess I'm modifying the entirety of the wireguard implementation so that it takes in esp_netif_t instead of the raw lwip netif, everywhere. That's going to take a while. |
|
Actually I don't even know if it's possible. A lot of the functions used depend on having direct access to information that just isn't exposed with the wrapped esp_netif_t |
I think I finally found an API to do something similar though the other way around. Read on.
The above is by design. Basically what you are observing is a C-style "visibility" trick utilized by ESP-IDF where they want to keep the layout of the Here's a link I found that should help get you started. So this way you But I think you need to modify the LwIP Wireguard impl a bit as it thinks it can "manually" create a raw LwIP netif, then wrap it as |
Yeah that's what I had in mind as well. I checked and we don't have that function, only those two pub fn esp_netif_get_netif_impl_name(esp_netif: *mut esp_netif_t, name: *mut ::core::ffi::c_char) -> esp_err_t;
pub fn esp_netif_get_netif_impl_index(esp_netif: *mut esp_netif_t) -> ::core::ffi::c_int;I'll try to make a PR for esp-idf-sys, we'll see where this goes. Thanks again for the help, I probably wouldn't be able to figure this out alone. |
|
But how do I create a (wrapped) wireguard interface? It requires a stack that I can't specify because the options don't fit having a virtual interface. This wasn't a problem with a raw lwip interface since all I needed was an ip configuration. pub struct esp_netif_config {
#[doc = "< base config"]
pub base: *const esp_netif_inherent_config_t,
#[doc = "< driver config"]
pub driver: *const esp_netif_driver_ifconfig_t,
#[doc = "< stack config"]
pub stack: *const esp_netif_netstack_config_t,
}extern const esp_netif_netstack_config_t *_g_esp_netif_netstack_default_eth;
extern const esp_netif_netstack_config_t *_g_esp_netif_netstack_default_br;
extern const esp_netif_netstack_config_t *_g_esp_netif_netstack_default_wifi_sta;
#ifdef CONFIG_ESP_WIFI_SOFTAP_SUPPORT
extern const esp_netif_netstack_config_t *_g_esp_netif_netstack_default_wifi_ap;
#endif
#ifdef CONFIG_ESP_WIFI_NAN_ENABLE
extern const esp_netif_netstack_config_t *_g_esp_netif_netstack_default_wifi_nan;
#endif
#ifdef CONFIG_PPP_SUPPORT
extern const esp_netif_netstack_config_t *_g_esp_netif_netstack_default_ppp;
#endifDo I need to write the functions myself? That's gonna be a huge pain I have no clue how to do this. |
|
Especially because sifting through the bridge code in lwip, I find this: if (!(portif->flags & NETIF_FLAG_ETHARP) || !(portif->flags & NETIF_FLAG_ETHERNET)) {
/* can only add ETHERNET/ETHARP interfaces */
return ERR_VAL;
}So unless I specify |
|
Well. This thread talks about it espressif/esp-idf#13600. I'll look into that. |
|
Okay so I was stupid I already have the functions necessary to create the netstack. In esp_wireguard.c: wg_netif = netif_add(
&wg_netif_struct,
ip_2_ip4(&ip_addr),
ip_2_ip4(&netmask),
ip_2_ip4(&gateway),
&wg, &wireguardif_init,
&ip_input);So if I just do static const struct esp_netif_netstack_config mysuperduperconfig = {
.lwip = {
.init_fn = wireguardif_init,
.input_fn = ip_input,
}
};While not forgetting to expose that config so that the rust layer can see it, then we might have something workable there... |
|
So this isn't quite working like I would want it to, as the above mentioned config have these types: typedef err_t (*init_fn_t)(struct netif*);
typedef esp_netif_recv_ret_t (*input_fn_t)(void *netif, void *buffer, size_t len, void *eb);
struct esp_netif_netstack_lwip_vanilla_config {
init_fn_t init_fn;
input_fn_t input_fn;
};While /** Function prototype for netif init functions. Set up flags and output/linkoutput
* callback functions in this function.
*
* @param netif The netif to initialize
*/
typedef err_t (*netif_init_fn)(struct netif *netif);
/** Function prototype for netif->input functions. This function is saved as 'input'
* callback function in the netif struct. Call it when a packet has been received.
*
* @param p The received packet, copied into a pbuf
* @param inp The netif which received the packet
* @return ERR_OK if the packet was handled
* != ERR_OK is the packet was NOT handled, in this case, the caller has
* to free the pbuf
*/
typedef err_t (*netif_input_fn)(struct pbuf *p, struct netif *inp);The init function seems transparent enough, so I think I can use |
|
Isn't there a way to make this easier for myself? The wiki says this:
The wireguard interface sets itself as the default interface, so shouldn't all traffic from ethernet naturally be forwarded to the wg netif by default without any intervention on my part? Yet, So either there is something I fundamentally misunderstand about forwarding or there's a bug somewhere. |
Ok so, having abandoned everything else as too complicated, I managed to get this to work somewhat. The problem was I was setting the wg_netif's ip to 0.0.0.0 and ip_forward checks that the ip is not 0.0.0.0 to send the packets to the default_netif. So going from PC -> ETH NETIF -> WG NETIF -> STA -> WG ENDPOINT works. But the other way doesn't because it stops at the WG NETIF. The packet gets decrypted and then doesn't know where to go so it's dropped. I'm having the exact same problem in reverse except this time I can't use the default netif stratagem. |
Hi,
I'm trying to wrap my head around how to route traffic from this netif to another? Since it's set as a dhcp router it doesn't seem like it's at all possible. I'd need to set the gateway of this netif to the ip of the other netif I want to route traffic through but that doesn't seem possible? The gateway here is only to set the ip of the netif itself inside its subnet from what I understand.
Is there a way to make the DHCP server run independently and have this configuration be a client instead?
The text was updated successfully, but these errors were encountered: