Saturday, May 3, 2014


A good idea with bad usage: /dev/urandom



Last week, I wrote two articles pointing out issues with unofficial porting efforts of LibreSSL. In these articles, I highlighted some issues that I currently see going on with some of these projects.

In the second article, I called attention to poor arc4random_buf() implementations being created, specifically saying: "Using poor sources of entropy like /dev/urandom on Linux, or worse, gettimeofday(), and using them to generate long-lived keys." Which seemed to irk some. Now for those that care to understand the quoted issue in its proper context should go look for the current LibreSSL porting projects they can find via Google, and review their source. However, most seem to be understanding this as some kind of argument about /dev/random versus /dev/urandom. It isn't. So without further ado:

Poorly seeding a cryptographically secure pseudorandom number generator (CSPRNG)

In order for many forms of cryptography to work properly, they depend on secrecy, unpredictability, and uniqueness. Therefore, we need a good way to use many unpredictable values.

Now randomness doesn't really exist. When we humans see something as random, it's only because we don't know or understand all the details. Therefore, any perceived randomness on your part is your inability to track all the variables.

Computers are very complex machines, where many different components are all working independently, and in ways that are hard to keep track of externally. Therefore, the operating system is able to collect variables from all the various hardware involved, their current operating conditions, how they handle certain tasks, how long they take, how much electricity they use, and so on. These variables can now be combined together using very confusing and ridiculous algorithms, which essentially throw all the data through a washing machine and the world's worst roller coaster. This result is what we call entropy.

Now entropy might not be that large, and can only provide a small amount of unique random values to play with. However, a small amount of random values can be enough to create trillions of pseudo-random values. To do so, one uses some of the aforementioned ridiculous algorithms to produce two values. One value is never seen outside the pseudo-random number generator, and is used as part of the calculations for the next time a random value is requested, and the other value is output as the pseudo-random value generated for a single request.




A construct of this nature can allow an unlimited amount of "random" values to be generated. As long as this technique never repeats and is unpredictable, then it is cryptographically secure. Of course since the algorithm is known, the entropy seeding it must be a secret, otherwise it is completely predictable.

Different algorithms have different properties. OpenBSD's arc4random set of functions are known to be able to create a very large amount of good random values from a very little amount of entropy. Of course the better entropy it is supplied with, the better it can perform, so you'll always want the best entropy possible. As with any random number generator, supply it with predictable values, and its entire security is negated.

arc4random and /dev/(u)random

So, how does one port the arc4random family to Linux? Linux is well known for inventing and supplying two default files, /dev/random and /dev/urandom (unlimited random). The former is pretty much raw entropy, while the latter is the output of a CSPRNG function like OpenBSD's arc4random family. The former can be seen as more random, and the latter as less random, but the differences are extremely hard to measure, which is why CSPRNGs work in the first place. Since the former is only entropy, it is limited as to how much it can output, and one needing a lot of random data can be stuck waiting a while for it to fill up the random buffer. Since the latter is a CSPRNG, it can keep outputting data indefinitely, without any significant waiting periods.

Now theoretically, one can make the arc4random_buf() function a wrapper around /dev/urandom, and be done with it. The only reason not to is because one may trust the arc4random set of algorithms more than /dev/urandom. In that case, would /dev/urandom be trusted enough to seed the arc4random algorithms, which is then in turn used for many other outputs, some of which end up in RSA keys, SSH keys, and so on? I'll leave that question to the cryptographic experts. But I'll show you how to use /dev/urandom poorly, how to attack the design, and corrective efforts.

First, take a look at how one project decided to handle the situation. It tries to use /dev/urandom, and in a worse case scenario uses gettimeofday() and other predictable data. Also, in a case where the read() function doesn't return as much as requested for some reason, but perhaps returned a decent chunk of it, the gettimeofday() and getpid() calls will overwrite what was returned.

This very much reminds me of why the OpenBSD team removed the OpenSSL techniques in the first place. You do not want to use a time function as your primary source of entropy, nor with range limited values, and other dumb things. If you're going to use time, at the very least use clock_gettime() which provides time resolution that is 1,000 times better than gettimeofday(), and can provide both current time, and monotonic time (time which cannot go backwards). Additionally, both gettimeofday() and clock_gettime() on 64-bit systems will return values where the top half of their data will be zeroed out, so you'll want to ensure you throw that away, and not just use their raw data verbatim. Further, how this one project uses /dev/urandom, like most other projects, is terrible.

Attacking /dev/(u)random usage

A naive approach to use /dev/urandom (and /dev/random) is as follows:

int fd = open("/dev/urandom", O_RDONLY);
if (fd != -1)
{
  uint8_t buffer[40];
  if (read(fd, buffer, sizeof(buffer)) == sizeof(buffer)))
  {
    //Do what needs to be done with the random data...
  }
  else
  {
    //Read error handling
  }
  close(fd);
}
else
{
  //Open error handling
}

This tries to open the file, ensures it was opened, tries to read 40 bytes of data, and continues what it needs to do in each scenario. Note, 40 bytes is the amount arc4random wants, which also happens to be 8 bytes more than the Linux manual page says you should be using at a time from its random device.

The first common mistake here is using read() like this. read() can be interrupted. Normally it can't be interrupted for regular files, but this device is not a regular file. Some random device implementations specifically document that read() can be interrupted upon them.

So now our second attempt:

//Like read(), but keep reading upon interuption, until everything possible is read
ssize_t insane_read(int fd, void *buf, size_t count)
{
  ssize_t amount_read = 0;
  while ((size_t)amount_read < count)
  {
    ssize_t r = read(fd, (char *)buf+amount_read, count-amount_read);
    if (r > 0) { amount_read += r; }
    else if (!r) { break; }
    else if (errno != EINTR)
    {
      amount_read = -1;
      break;
    }
  }
  return(amount_read);
}

int success = 0;
int fd = open("/dev/urandom", O_RDONLY);
if (fd != -1)
{
  uint8_t buffer[40];
  ssize_t amount = insane_read(fd, buffer, sizeof(buffer)); //Grab as much data as we possibly can
  close(fd);

  if (amount > 0)
  {
    if (amount < sizeof(buffer))
    {
      //Continue filling with other sources
    }
    //Do what needs to be done with random data...
    success = 1; //Yay!
  }
}

if (!success)
{
  //Error handling
}

With this improved approach, we know we're reading as much as possible, and if not, then we can try using lesser techniques to fill in the missing entropy required. So far so good, right?

Now, let me ask you, why would opening /dev/(u)random fail in the first place? First, it's possible the open was interrupted, as may happen on some implementations. So the open() call should probably be wrapped like read() is. In fact, you may consider switching to the C family fopen() and fread() calls which handle these kinds of problems for you. However, opening could be failing because the application has reached its file descriptor limit, which is even more prevalent with the C family of file functions. Another possibility is that the file doesn't even exist. Go ahead, try to delete the file as the superuser, nothing stops you. Also, you have to consider that applications may be running inside a chroot, and /dev/ entries don't exist.

I'll cover some alternative approaches for the above problems later. But if you managed to open and read all the data needed, everything is great, right? Wrong! How do you even know if /dev/(u)random is random in the first place? This may sound like a strange question, but it isn't. You can't just trust a file because of it's path. Consider an attacker ran the following:

void sparse_1gb_overwrite(const char *path)
{
  int fd;
  char byte = 0;
  //No error checking
  unlink(path);
  fd = open(path, O_CREAT | O_WRONLY | O_TRUNC, 0644);
  lseek(fd, 1073741822, SEEK_SET);
  write(fd, &byte, 1);
  close(fd);
}

sparse_1gb_create("/dev/random");
sparse_1gb_create("/dev/urandom");

Now both random devices are actually large sparse files with known data. This is worse than not having access to these files, in fact, the attacker is able to provide you with a seed of his own choosing! A strong cryptographic library should not just assume everything is in a proper state. In fact, if you're using a chroot, you're already admitting you don't trust what will happen on the file system, and you want to isolate some applications from the rest of the system.

So the next step is to ensure that the so called device you're opening is actually a device:

int success = 0;
int fd = insane_open("/dev/urandom", O_RDONLY);
if (fd != -1)
{
  struct stat stat_buffer;
  if (!fstat(fd, &stat_buffer) && S_ISCHR(stat_buffer.st_mode)) //Make sure we opened a character device!
  {
    uint8_t buffer[40];
    ssize_t amount = insane_read(fd, buffer, sizeof(buffer)); //Grab as much data as we possibly can

    if (amount > 0)
    {
      if (amount < sizeof(buffer))
      {
        //Continue filling with other sources
      }
      //Do what needs to be done with random data...
      success = 1; //Yay!
    }
  }
  close(fd);
}

if (!success)
{
  //Error handling
}


So now we're out of the woods, right? Unfortunately not yet. How do you know you opened the correct character device? Maybe /dev/urandom is symlinked to /dev/zero? You can run lstat() on /dev/urandom initially, but that has TOCTOU issues. We can add the FreeBSD/Linux extension O_NOFOLLOW to the open command, but then /dev/urandom can't be used when it's linked to /dev/random as it is on FreeBSD, or linked to some other location entirely as on Solaris. Furthermore, avoiding symlinks is not enough:

void dev_zero_overwrite(const char *path)
{
  unlink(path);
  mknod(path, S_IFCHR | 0644, makedev(1, 5));
}

dev_zero_overwrite("/dev/random");
dev_zero_overwrite("/dev/urandom");

If an attacker manages to run the above code on Linux, the random devices are now both in their very essence /dev/zero!

Here's a list of device numbers for the random devices on various Operating Systems:




Linux FreeBSD DragonFlyBSD NetBSD OpenBSD Solaris
/dev/random 1:8 0:10 8:3 46:0 45:0 0:0
/dev/urandom 1:9

8:4 46:1 45:2 0:1
/dev/srandom







45:1

/dev/arandom







45:3


If your application is running with Superuser privileges, you can actually create these random devices anywhere on the fly:
int result = mknod("/tmp/myurandom", S_IFCHR | 0400, makedev(1, 9)); //Create "/dev/urandom" on Linux

Of course, after opening, you want to ensure that you're using what you think you're using:

int success = 0;
int fd = insane_open("/dev/urandom", O_RDONLY);
if (fd != -1)
{
  struct stat stat_buffer;
  if (!fstat(fd, &stat_buffer) && S_ISCHR(stat_buffer.st_mode) &&
      ((stat_buffer.st_rdev == makedev(1, 8)) || (stat_buffer.st_rdev == makedev(1, 9)))) //Make sure we opened a random device
  {
    uint8_t buffer[40];
    ssize_t amount = insane_read(fd, buffer, sizeof(buffer)); //Grab as much data as we possibly can

    if (amount > 0)
    {
      if (amount < sizeof(buffer))
      {
        //Continue filling with other sources
      }
      //Do what needs to be done with random data...
      success = 1; //Yay!
    }
  }
  close(fd);
}

The Linux user manual page for the random devices explicitly informs the reader of these magic numbers, so hopefully they won't change. I have no official sources for the magic numbers on the other OSs. Now, you'll notice that I checked here that the file descriptor was open to either Linux random device. A system administrator may for some reason replace one with the other, so don't necessarily rely on a proper system using the expected device under the device name you're trying to use.

This brings us back to /dev/random versus /dev/urandom. Different OSs may implement these differently. On FreeBSD for example, there is only the former, and it is a CSPRNG. MirBSD offers five different random devices with all kinds of semantics, and who knows how a sysadmin may shuffle them around. On Linux and possibly others, /dev/urandom has a fatal flaw that in fact it may not be seeded properly, so blind usage of it isn't a good idea either. Thankfully Linux and NetBSD offer the following:

int data;
int result = ioctl(fd, RNDGETENTCNT, &data); //Upon success data now contains amount of entropy available in bits

This ioctl() call  will only work on a random device, so you can use this instead of the fstat() call on these OSs. You can then check data to ensure there's enough entropy to do what you need to:

int success = 0;
int fd = insane_open("/dev/urandom", O_RDONLY);
if (fd != -1)
{
  uint8_t buffer[40];
  int entropy;
  if (!ioctl(fd, RNDGETENTCNT, &entropy) && (entropy >= (sizeof(buffer) * 8))) //This ensures it's a random device, and there's enough entropy
  {
    ssize_t amount = insane_read(fd, buffer, sizeof(buffer)); //Grab as much data as we possibly can

    if (amount > 0)
    {
      if (amount < sizeof(buffer))
      {
        //Continue filling with other sources
      }
      //Do what needs to be done with random data...
      success = 1; //Yay!
    }
  }
  close(fd);
}

However, there may be a TOCTOU between the ioctl() and the read() call, I couldn't find any data on this, so take the above with a grain of salt. This also used to work on OpenBSD too, but they removed the ioctl() command RNDGETENTCNT a couple of versions back.

Linux has one other gem which may make you want to run away screaming. Look at the following from the manual for random device ioctl()s:

RNDZAPENTCNT, RNDCLEARPOOL
Zero the entropy count of all pools and add some system data (such as wall clock) to the pools.
If that last one does what I think it does, is any usage ever remotely safe?

/dev/(u)random conclusion


After all this, we now know the following:
  • This is hardly an interface which is easy to use correctly (and securely).
  • On some OSs there may not be any way to use it correctly (and securely).
  • Applications not running with Superuser privileges may have no way to access the random device.
  • An application at its file descriptor limit cannot use it.
  • Many portability concerns.

For these reasons, OpenBSD created arc4random_buf() in the first place. It doesn't suffer from these above problems. The other BSDs also copied it, although they may be running on older less secure implementations.

Alternatives


In addition to a CSPRNG in userspace, the BSDs also allow for a way to get entropy directly from the kernel:

#define NUM_ELEMENTS(x) (sizeof(x)/sizeof((x)[0]))
uint8_t buffer[40];
size_t len = sizeof(buffer);
int mib[] = { CTL_KERN, KERN_RND };
int result = sysctl(mib, NUM_ELEMENTS(mib), buffer, &len, 0, 0);


KERN_RND may also be replaced with KERN_URNDKERN_ARND, and others on the various BSDs. FreeBSD also has a similar interface to determine if the kernel's CSPRNG is properly seeded, see the FreeBSD user manual page for more details. DragonFlyBSD also provides read_random() and read_random_unlimited() as direct no-nonsense interfaces to the underlying devices for /dev/random and /dev/urandom.

Now Linux used to provide a similar interface as the BSDs:
#define NUM_ELEMENTS(x) (sizeof(x)/sizeof((x)[0]))
uint8_t buffer[40];
size_t len = sizeof(buffer);
int mib[] = { CTL_KERN, KERN_RANDOM, RANDOM_UUID };
int result = sysctl(mib, NUM_ELEMENTS(mib), buffer, &len, NULL, 0);

However, this interface was removed a couple of versions back. Kind of makes you wonder why there is no sane simple to use standardized API everywhere that doesn't depend on a house of cards. This is the kind of thing you would think would be standardized in POSIX.

Conclusion


There does not seem to be any good cross-platform techniques out there. Anything done with the current technology will probably require a ton of conditional compiles, annoying checks, and ridiculous workarounds. Although, there should be enough basis here to come up with something with a high confidence level. It'd be interesting to see what the LibreSSL team (or the OpenSSH team) comes up with when they get around to porting what's needed here. As I said before, until then, avoid the non-official ports out there, which use poor sources of entropy like an insecure use of /dev/urandom on Linux, which then falls back on gettimeofday(), and is used to generate long-lived keys.

211 comments:

1 – 200 of 211   Newer›   Newest»
Max Lybbert said...

I wasn't expecting this post so quickly. Thank you.

I originally blew off the concerns about an attacker monkeying with /dev/urandom because an attacker would need to have administrative privileges to do so, and if they have that they can already do a lot more damage (Raymond Chen refers to this as requiring the attacker to be on the other side of an airtight hatch: https://www.google.com/search?q=airtight+hatch+oldnewthing ). However, the machine you're one may not be the attacker's final target, your SSL keys might be (or, rather, your customers' private data might be the final target, and the attacker hopes to get that data through administrative rights on the machine).

henke37 said...

Next up, why you shouldn't use the unpriviledged assembly instruction to load random data.

insane coder said...

Max: I was already asked to write this yesterday, and was half way done by the time you wrote your first comment. So yeah, it seems quick :)

Now, regarding your points:
Not necessarily an attacker either, The NetBSD manpage *recommends* you shuffle around your random devices to what you think is best. If you program to expect one design and get the other, you're going to get problems.

And don't forget that it may be seeded incorrectly. Thankfully you can use the ioctl() to check for that.

In any case, one shouldn't make assumptions in secure programs, they should be verified, and (/dev/urandom || gettimeofday()) is a very poor source of entropy. The first may fail for a multitude of reasons, and the last one, especially when done terribly, is a joke.

insane coder said...

Another issue with the entire design of the Linux random devices is that even a regular user can run: cat /dev/random > /dev/null & cat /dev/zero > /dev/random

If that's running, entropy will be degraded.

>sysctl kernel.random
kernel.random.poolsize = 4096
kernel.random.entropy_avail = 12

12 bits of entropy is not a good situation to be in...

busterb said...

Hi, thanks for the feedback insane coder. I had posted this version more as a PoC to the OpenSSH mailing list, you can follow the discussion there: http://www.gossamer-threads.com/lists/openssh/dev/58537

I'll note your comments on the project issue tracker so that they're not lost.

insane coder said...

Hi Unknown,

I'm not saying you should follow any of these blindly, but take a look at:
https://github.com/libevent/libevent/blob/master/arc4random.c
https://github.com/nmathewson/libottery/blob/master/src/ottery_entropy_urandom.c

For explicit_bzero(), you might want to consider:
void explicit_bzero(void *p, size_t n) __attribute__((optimize("O0")))
Or: http://cvsweb.netbsd.org/bsdweb.cgi/src/common/lib/libc/string/Attic/explicit_bzero.c?rev=1.1&content-type=text/x-cvsweb-markup&only_with_tag=MAIN

Some platforms have memset_s(), and Microsoft has SecureZeroMemory().

dlg said...

there's no TOCTOU problem between an ioctl and a read like you describe unless you're in a threaded program and the other thread is doing things you're really unaware of.

ps, dont use threads.

insane coder said...

dlg: How do you know another process won't be using the device at the same time and alter the value returned from the ioctol() before the read()?

Based on some quick testing on Linux, that very much seems like a possibility. Other OSs may act differently.

dlg said...

ah, i thought you were asking if the device an fd was referencing could change between ioctl and read.

the state of the device can obviously change between syscalls unless it makes attempts to be opened by only one thing at a time, and isn't affected by something like hotplug or interrupts. however dup and fork kind of ruin that too.

insane coder said...

According to manpages on other platforms, each process trying to open the random device gets a unique random device, so there would be no race condition on them.

I don't really understand what Linux's random "API" is trying to accomplish. It is very hard to use correctly, and some of its features just seem useless or deceptive.

Unknown said...

Some of your statements contradict the following article:
Myths about urandom

IMHO, an initial seed of 128 bit should be enough to produce random numbers forever, no need for reseeding. So /dev/urandom is not less secure than /dev/random.

May you comment on this issue?

insane coder said...

Hello Dobin,

Where did I say /dev/urandom is less secure than /dev/random?

Unknown said...

Hmm after reading the article again, it seems you did not explicetly state this fact :-)

But:
So, how does one port the arc4random family to Linux? Linux is well known for inventing and supplying two default files, /dev/random and /dev/urandom (unlimited random). The former is pretty much raw entropy, while the latter is the output of a CSPRNG function like OpenBSD's arc4random family

The thing with "pretty much raw entropy" caught my eye. According to the mentioned article, /dev/random is the same as /dev/urandom, but will block if the super-mega-magic entropy-measurement function thinks it does not have enough entropy.

insane coder said...

Hi Dobin,

I tried giving an explanation which should be simple for a layman to follow of what goes on. I wouldn't take it as precise fact, especially when there's a washing machine involved in the explanation ;)

The issues between /dev/random and /dev/urandom is slightly more complex than just blocking. How does /dev/urandom keep supplying an unlimited about of data? Will its algorithm ever start becoming predictable? How does this interact with seeding the algorithm in arc4random, which also reseeds from time to time?

As is well known, the primary issue with /dev/urandom is when you use it in a library which uses it once to seed, and is started at bootup, say within Apache. To combat the issues there, 1) I showed that the ioctl() can be used to see if /dev/urandom has entropy (whether that works properly or not is another matter). 2) I suggested that arc4random_buf() should in fact be a direct wrapper around /dev/urandom, so you never have to worry about some initial poor seed infecting an application's entire life time, as you'll just call it directly each time you need some random.

Unknown said...

The RANDOM_UUID sysctl wasn't removed, but the syscall mode of access was simply made optional with the kernel configuration option CONFIG_SYSCTL_SYSCALL.

Ubuntu 14.01 still builds with sysctl syscalls, and so the method I posted still works, while Red Hat apparently removed them awhile ago. I haven't tested other distributions.

This is pretty lame on the part of Red Hat, because quite a few applications use the RANDOM_UUID method. You can still, for example, cat /proc/sys/kernel/random/uuid, you just cannot access it without /proc.

Here's a choice quote from the Tor source code:

/* On linux, sysctl is deprecated. Because proc is so awesome that you shouldn't _want_ to write portable code, I guess? */

Unknown said...

Hey Mr. Insane

I also dont know if the normal /dev/urandom is a cryptographically secure PRNG. It should be ;-)

1) To seed arc4random with /dev/[u]random, it's like:
"randomness from measuring IRQ timings" -> (/dev/[u]random) PRNG -> seed arc4random

2) And the better way:
"randomness from measuring IRQ timings" -> seed arc4random

Is 1) really much worse than 2? Can't answer that too.


Nevertheless, good writeup about the difficulties in "just acquiring" the random numbers by userspace code in hostile environment :-)

insane coder said...

Hi Dobin,

Any randomness should be first ran through some algorithm to remove biases that a single source may carry.

If you're very interested in this, you may want to read:
http://eprint.iacr.org/2013/338.pdf
http://www.oreilly.com/catalog/secureprgckbk/chapter/ch11.pdf (this covers real world use too!)

There's also been some talk lately about a backdoor in Linux's RNG, read through some commit messages here: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/log/drivers/char/random.c?showmsg=1

Some random devices may not too secure, check out /dev/srandom and /dev/prandom on MirBSD. This page covers a lot of interesting material regarding different devices on different OSs and testing: https://calomel.org/entropy_random_number_generators.html

Lastly, you're assuming that arc4random is secure with a good random seed. I personally haven't seen any papers on its strengths and weaknesses, or how it compares to alternatives. I'd love if someone who really knows their stuff would determine how arc4random stacks up to Linux's method, and the pros and cons of one seeding the other. If arc4random is truly a real improvement that can be applied after a /dev/urandom step, then there's no reason why it shouldn't be added as the final pass to the output of /dev/(u)random itself.


I'm glad you enjoyed my article.

Unknown said...

"Why doesn't linux have a sane API" - the same reason linux doesn't have a lot of sane API's: Usually, it looks like this.

insane coder said...

Hi Bob,

In order for Linux to have a sane API, it would have to provide a sane interface for the C library to then provide a sane interface to the application developers.

The patch there is solely for glibc, and makes all the same mistakes this article is rallying against. I would reject it too.

insane coder said...

Since enough people were writing to me about /dev/random vs. /dev/urandom, I wrote an article about it an other practical problems.

Joseph M. Schwartz said...

Hey, have you seen http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libcrypto/crypto/

Do you think they handle entropy properly?

insane coder said...

Hi Joseph,

Yet, I saw that a couple of days ago.
Note, that is being constantly worked on, so my remarks may no longer apply by the time you read this.

1) It seems they noted some of the information presented here, so that's good.

2) They seem not to handle some issues mentioned here, or in my other article.

Notably, they're relying on /dev/urandom to be returning good data, which may not be true close to system start up. The approach they're taking if reading /dev/urandom fails should really be used in addition to /dev/urandom, not instead of. Combining data from all over (hashing it all together) will serve to improve the basic situation. Of course there is still the question as to whether not being able to combine in /dev/(u)random as one of the sources should result in an abort.

They're using the sysctl() call on Linux, which doesn't exist. I don't know which distribution has an actually system call there (even with what they're doing), but not a single one of my servers, which have multiple distros, some even 5 years old, supports it. Even if a distro does offer that function, I'm not sure it does what they think it does.

Since I don't have access to a system which supports it, I haven't actually tested the function to see what it returns. But based on some of the information I've read about it, it may only return a limited amount of data, and it won't be in the format you're expecting. Meaning, you really need to check how much you're receiving, and loop if need be. Also, you'll have to convert the encoding of the data to raw binary.

This may also be entirely moot, because if the function does work, it may very well just wrap to file system reads, in which case you're not really gaining anything, and you surely cannot trust the data you receive from it.

However, you may be able to mount proc in cases where you cannot mount devtmpfs (or whatever flavor alternate is available), in which case it can be worthwhile to read it via the file system directly, perhaps with some preads to ensure successive calls are returning different results.

3) There's a lot more they can be doing to gather entropy, especially for bad situations. As I described in my other article, grabbing hardware serial numbers, perhaps from the network devices can help some situations. SIOCGIFCONF and friends can help, and on some OSs, that will also return some real entropy from the network devices. There's getifaddrs() too, although that can be error prone if you're not careful, and you really need to test how you're parsing the returned data if certain whacky network devices are present (with SIOCGIFCONF, you can just hash the buffer as a whole, and the only parsing you need is perhaps for simple SIOCGIFHWADDR calls on OSs where available).

On x86 CPUs, the cpuid instruction can also return some unique to processor details, and on modern ones, entropy as well, as some cpuid calls return details about temperature and power.

4) Taking the address of main is a bad idea. Depending how the library is linked/loaded, that's going to fail.

5) Based on the idea of taking addresses, sbrk(0) should be mixed in. Address of environ can be good too. Linux has 3 pointers, etext, edata, end, on Solaris, _etext, _edata, _end, which point to different areas of the memory layout should be mixed in too. Some other C functions may also be in separate libraries and can move around a bit, for example, malloc, clock_gettime, and pthread_mutex_lock, so those are good in addition to printf.

insane coder said...

6) Regarding time with clock_gettime(), I use the following in my libraries:

#ifdef CLOCK_REALTIME_PRECISE
TIME_UPDATE(CLOCK_REALTIME_PRECISE)
#else
TIME_UPDATE(CLOCK_REALTIME) //Everyone who has clock_gettime() has CLOCK_REALTIME
#endif

#ifdef CLOCK_HIGHRES
TIME_UPDATE(CLOCK_HIGHRES)
#elif defined(CLOCK_MONOTONIC_RAW)
TIME_UPDATE(CLOCK_MONOTONIC_RAW)
#elif defined(CLOCK_MONOTONIC_PRECISE)
TIME_UPDATE(CLOCK_MONOTONIC_PRECISE)
#elif defined(CLOCK_MONOTONIC)
TIME_UPDATE(CLOCK_MONOTONIC)
#endif

So if clock_gettime() is present, it'll use the most accurate real time, and for those also supporting monotonic time, the most accurate one of that. Of course their approach of mixing in multiple of these is good, but they're not always hitting some of the more important ones.

7) The use of gettimeofday() when you have clock_gettime() is pretty pointless. However, alongside clock_gettime() of day, it can be useful to use clock_getres() for each timer, as those will differ system to system.

8) times() may have other useful data to mix in.

9) On Linux, there's probably a few more getauxval() values that can be useful to mix in. A fallback for implementing that function for older GLIBCs wouldn't be a bad idea either.

Okay, I think that pretty much wraps it up.

Joseph M. Schwartz said...

Wow, I wasn't expecting you to be so thorough. Thank you!

You mentioned you have your own libraries, is there anything you do in your libraries you have not covered?

If all your suggestions are implemented, would you yourself use the LibReSSL RAND API?

insane coder said...

Hi Joseph,

>You mentioned you have your own libraries, is there anything you do in your libraries you have not covered?

All the important things I do is more or less covered in the articles here, and the resources I link to.

As far as collecting entropy goes, the biggest difference between my implementation and LibreSSL's is that I have a list of the random device numbers for OSs where they do not fluctuate (the BSDs fluctuate), and on some OSs, use their APIs for getting the device numbers (for example NetBSD provides such an API). These device numbers are used to verify the devices being opened, and also attempt to mknod() them if they do not exist in the usual location.

For the OSs that support it, I also attempt to mount devfs or devtmpfs.

>If all your suggestions are implemented, would you yourself use the LibReSSL RAND API?

No, I wouldn't.

Today, with the proliferation of virtual machines, especially for hosting, traditional random APIs no longer cut it. As I mentioned in my other article, virtual machines can be rolled back to a particular state, or are sometimes cloned from the same state. These things happen in real hosting environments in response to malfunctions, attacks, or heavy load.

Mixing in the current time (and perhaps MAC address) into the entropy upon the initial machine boot is crucial (see my previous response regarding combining /dev/(u)random with the other functions). Now while that will guard against several problems, it wouldn't help against live state loading, nor cases where the VM hasn't updated the real time yet.

Buffered APIs like arc4random*() and RAND_*() layered over it will be generating identical data in these situations, and can be attacked. Mixing in the current time and any hardware RNGs if available on every use can help, but you really need to be using hedged cryptography.


The APIs provided by OpenBSD and LibreSSL also don't cover all the scenarios I need them to. For example, I sometimes need a uniform number of a particular type within a particular range, and arc4random_uniform() doesn't actually provide uniform results, doesn't simply allow for ranges, and only works with a single type.

When using a standalone command to generate RSA keys, I'd also prefer if it blocked on /dev/random, but this point is debatable.

Joseph M. Schwartz said...

"but you really need to be using hedged cryptography"

What is that? I do not see you talking about this anywhere.

"arc4random_uniform() doesn't actually provide uniform results"

What do you mean? I see this in the manual "arc4random_uniform() will return a uniformly distributed random number less than upper_bound. arc4random_uniform() is recommended over constructions like “arc4random() % upper_bound” as it avoids "modulo bias" when the upper bound is not a power of two."

"only works with a single type"

So what? You can easily extend the functions to other types as needed.

insane coder said...

Hi Joseph,

>"but you really need to be using hedged cryptography"

>What is that? I do not see you talking about this anywhere.

I mentioned the overall idea of it, and link to the paper in the comments of my other article.

There's also a video at Microsoft Research, although the site doesn't seem to be loading for me at the moment. I got a direct link to the MP4 out of Google's cache.

The essential of the idea is that you should ensure the randomness generated is generated specifically for whatever it is you're generating it for. So for example, if you're generating a random temporary password for a user, you mix the user's ID or user's name into the entropy pool before obtaining the random data. If you're generating a temporary symmetric encryption key for a remote connection, you can mix in the IP address, and so on. This ensures that if two different users or machines or whatever are working off the same state, they'll still get different results between the two of them.

This will need a slightly different API, and applications have to make use of it where applicable, but OpenBSD doesn't even provide an underlying API to accomplish this. They used to have an arc4random_addrandom() function which could theoretically get the job then, but they since removed it. In any case, that API was poorly designed, the hedged randomness function should combine the various functions as needed internally to prevent misuse, and ensure there aren't any gotchas occurring between the two calls.

>I see this in the manual "arc4random_uniform() will return a uniformly distributed random number less than upper_bound. arc4random_uniform() is recommended over constructions like “arc4random() % upper_bound” as it avoids "modulo bias" when the upper bound is not a power of two."

Yes, but that's not actually true. Like most naive implementations out there, the results aren't uniform and contain bias. Test it yourself if you don't believe me.

insane coder said...

>So what? You can easily extend the functions to other types as needed.

You just condoned the worst sin in all programming, code repetition.

As you've probably seen in the LibreSSL changelogs, OpenSSL was filled with cases where a bunch of code could be replaced with a single call to asprintf(). Now it's easy to write your own asprintf(), although there could be gotchas involved as I demonstrated. Functions make the code clearer, and ensure you don't screw up, or if you do, you only need to fix it once.

Unfortunately, with the UNIX principal of do just one thing and do it well, there's a severe lack of higher level functions in the various APIs. Sure, every now and then someone invents a higher level function like asprintf(), but for the most part, you find UNIX libraries filled with code repetition, and OpenBSD is no exception.

Take this example, which I noticed someone pointed out on OpenSSL Valhalla Rampage, see: http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libssl/src/crypto/engine/eng_list.c
In revision 1.11, issetugid() was placed before every getenv() call throughout the library. This kind of usage already indicates there should be a combined function. If not in the C library, at least internal to LibreSSL itself. In fact, GLIBC already noticed this issue is frequent and supplies a function secure_getenv().
Now look at revision 1.12, theo had to correct a mistake he made in using these two functions separately. That would likely not have happened if there was a combined function call available.

POSIX is finally starting to get it, and is adding more functions for atomicity reasons. Unfortunately this messed up development style and the ensuing issues and complications is quite prevalent throughout OpenBSD and other projects. As I mentioned in this article, read() is problematic, and generally needs to be wrapped in checks for EAGAIN, EINTR, and general looping. Yet these projects generally just employ the same code repetition everywhere leading to ugly bloated code and potential problems.

I'm all for not complicating the underlying C library, and providing new functions which may have questionable APIs ending up in public use. But something of LibreSSL's nature should have its own internal higher level functions to use wherever needed. I'd welcome another several thousand line source reduction by doing so.

insane coder said...

Besides different types for different sized integers, my random library also includes functions like these:
void rand_text_hedge_whitelist(char *buffer, size_t buffer_length, const void *hedge, size_t hedge_length, const char *whitelist);
void rand_text_hedge_vec_whitelist(char *buffer, size_t buffer_length, const struct iovec *hedge_iov, size_t iov_amount, const char *whitelist);

Which can be used for generating temporary passwords using certain allowed characters, ensure the returned random string is hedged, and null terminated.

Now such functions could easily be built from more primitive functions, but we want to ensure our developers use these things correctly, and provide higher level functions for both ease and correctness. It's a good thing too, because shortly after creating these, a coworker of mine pointed out how these two functions had a subtle bug due to the interactions of the primitives used to build them. I'm glad I only had to fix that in one place, and not throughout a ton of code recreating these.

To give you an idea of what I'm talking about here, my random API has over 70 functions. My string APIs (which go way beyond the silly things like strcpy(), strncpy(), strlcpy()) has several hundred functions. With consistent naming and APIs which were well thought out, developers also remember the different ones, and how they work and don't need to keep looking them up. More than once, one of our developers came across a certain situation where they needed some particular string handling function, just typed in what they thought it would be called if it existed along with its parameters, and figured they'd go add it to our massive list, but turns out it was already put there by a previous developer, along with the API they thought it would have. Now that's intuitive and useful, but unfortunately rarely exists, and all too often, a lot of functions leads to confusing misdesigned APIs. So don't expect such things to work in anything as messed up as a public C library. More often than not, in such cases, you end up with the garbage Microsoft provides.

Joseph M. Schwartz said...

Thanks for the information.
Hedged cryptography sounds like a good idea.

How can I test arc4random_uniform() for bias?

Your other points make sense in theory, but I think you're going a wee bit overboard.

insane coder said...

Hi Joseph,

>How can I test arc4random_uniform() for bias?

Rip out the random function and replace with one of your own which returns 0, then 1, then 2, etc...

Replace the loop with a failure mode.

Collect statistics on returned data.


Perhaps I should write an article about this.

Anonymous said...

Note: I'm posting this from my business account because I can apparently only post with a Google account here, but my employer has nothing to do with what I write here. It's not that I can foresee any conflict of interest, but you can never be too careful with these things.

> Rip out the random function and replace with one of your own which returns 0, then 1, then 2, etc...
>
> Replace the loop with a failure mode.
>
> Collect statistics on returned data.

Assuming I understood you correctly, I did exactly that for a few different upper bounds (not exceeding 42) and found no differences between the expected and actual distributions. Could you clarify when the problem occurs? Does it only happen with large upper bounds? Could you post some sample code, perhaps? I've studied this function in serious detail and am absolutely convinced that it's 100% correct, so your claims that it does the wrong thing have me puzzled. Also, if this is (in your words) a naive implementation, what would a proper implementation look like?

insane coder said...

Hello Jorden,

I modified the function to use a uint8_t instead of uint32_t, upper bound of 255 (254 really the way it works), and collected data. Suffice it to say, it fails, although performed better than some other approaches I saw.

I can't comment on using it with 42, I'm not aware of any native integer size which provide a limit of that number.

The function is incorrect, because the distribution is not uniform. For every single upper bound, the amount of hits for each number below it should be equal to every other number. For the maximum, the value of every number should be exactly 1. For some values for various upper bounds, you'll have results which do not match that.

If in proper testing, it did work correctly, perhaps that was of a version other than the ones I looked at. What I said certainly held true of the latest version in CVS at the time I wrote the above. I can double check with latest CVS again when I get a chance.

If need be, I can write an article about proper testing and implementation.

Anonymous said...

That function did not change in the OpenBSD tree since 2012, although recently it was moved to a separate file, without any changes to the actual code.

I did not change the return type of the random function, but 42 was the largest value of the upper bound argument I supplied to the uniform distribution function. Larger values require a larger array to store the results and more time to verify them, so there is an upper limit to the values that are feasible (although it's obviously a lot higher than 42).

I you changed the type to uint8_t, you may have run afoul of integer promotions. I would check that first if I were you. Of course, if you post your code here, everyone can take a look at it to see what your approach was.

insane coder said...

Hello Jorden,

I did not change the return type of the function. I redid the logic with uint8_t to test it, as testing a uint32_t would require more resources that I was willing to test with.

If one is not going to do an exhaustive test of the logic, it can't be verified to be correct.

As I said before I can write an article about how to handle this sort of thing properly.

Anonymous said...

Well, I feel like we're going in circles. Yes, it might be interesting to write an article about how to handle this properly, but in the meantime there's a more immediate issue to take on.

You wrote that arc4random_uniform is not implemented correctly. If this is true, it's a serious problem that needs to be solved immediately, given that this function has become something of an informal standard that half the world is now using. However, if it's not true, we need to clear that up, so as not to cast unnecessary doubt on a function that's actually correct.

I agree that only an exhaustive test can serve as proof that a function is correct. In fact, even an exhaustive test isn't necessarily sufficient, given that there are differences between platforms, compilers, etc. I did not claim that I performed an exhaustive test, but I did try to keep the alterations to the function minimal. I was already convinced that the function was correct based on an in-depth analysis of its code, and running the test (based on my initial understanding of what you did) reassured me. It is, however, still theoretically possible that I'm wrong and that the function is defective. Given that I was the one who suggested the latest change to this function, after MUCH deliberation, I'd like to consider myself something of an expert ("in a very narrow field", to quote a recently released humorous movie about being the only engineer in business meetings). That's why I believe I may know the explanation for your test results.

What you seem to have done (again, this is based on my understanding of what you wrote here, so please do correct me if I'm wrong) is to replace uint32_t with uint8_t and test that version exhaustively. While you may have performed an exhaustive test on your altered version, you did not test the original function exhaustively, nor did you technically test the original function AT ALL. This is important, because it means that conclusions about the correctness of the altered function (no matter if they are positive or negative) do not necessarily carry over to the original function.

Allow me to elaborate. There's an implicit assumption in the current code that uint32_t is not a candidate for integer promotions, i.e. that the arithmetic will be carried out in 32 bits, not in 64 bits or some other number of bits that is not 32. This assumption holds for all platforms on which OpenBSD runs, and for the platforms that have imported this code it probably holds as well. However, similar assumptions do not hold for uint8_t. Such values will be promoted to 32 bits (signed, probably) and that will affect the result. So, if you replaced all occurrences of uint32_t with uint8_t, you broke the code. Results obtained by running the altered function will be incorrect. However, this does not say anything at all about the correctness of the original function.

Will you help me confirm my hypothesis? If so, and if my analysis of how you performed the tests is correct, then please correct your altered version and report back on your findings.

All you need to do is take this one line:

min = -upper_bound % upper_bound;

And then replace it by these two:

min = -upper_bound;
min %= upper_bound;

insane coder said...

Hi Jorden,

I saw what you were saying and looked over my function again, and I noticed I placed my cast around the wrong segment, so I'm entirely to blame here.

I now plugged this function into my C++ uniform testing framework:

uint32_t uniform_openbsd(uint8_t upper_bound, uint8_t selected_rand)
{
if (upper_bound < UINT8_MAX)
{
++upper_bound;
if (upper_bound >= 2)
{
uint8_t min = static_cast<uint8_t>(-upper_bound) % upper_bound;
if (selected_rand < min) { return(UINT32_MAX); }
selected_rand %= upper_bound;
}
else { selected_rand = 0; }
}
return(selected_rand);
}

And results are perfectly uniform for every possible combination.

My testing framework includes many uniform functions found in various libraries and books (most of them considerably easier to correctly port to uint8_t and this API than OpenBSD's method). Nearly every implementation I've ran across fails an exhaustive test (and nearly every port was double and triple checked for correctness), and it seems in my haste to recently add OpenBSD's to it, I mistakenly casted the wrong component, and assumed that like many other implementations it carried similar mistakes.

I'm sorry for the false alarm, and can now confirm with exhaustive tests that OpenBSD is one of three implementations that I know of that are correctly implemented.

Anonymous said...

Ah, that's definitely good news. Thank you for admitting your mistake.

What are the other two correct implementations, by the way?

insane coder said...

Hello Jorden,

>Ah, that's definitely good news. Thank you for admitting your mistake.

If one cannot admit when they make a mistake, they should not be allowed to write anything.

>What are the other two correct implementations, by the way?

The other implementations are libottery and a corrected version I wrote based upon the one found in the Secure Programming Cookbook (In the SPC itself, it mistakes > for >=, and mishandles the case where the specified range is the entire range of possible values, but fixing it is easy).

Since I now have 3 working implementations, I ran an additional statistic regarding reloop, and noticed that OpenBSD unlike the other two has optimal relooping in all scenarios. The other two both cut off a whole set of random values when the range is a power of 2 minus 1. Meaning their retry random value is a bit worse than 50% on power of 2 minus 1. Nothing terrible, but can be slightly slower to find a random value in those cases.

Additionally, OpenBSD and my corrected SPC implementation has a faster algorithm than libottery, as the last division operation for the formers is only ran when the value is valid, whereas the latter need to run a division upon each random value received, even if thrown away.

Anonymous said...

Interesting stuff. It looks like there would be more than enough material for a real article, if you can find the time for it. This subject is still poorly understood by many, but the awareness is growing. A whole generation of programmers has grown up with the modulo trick, without realizing that it's subtly (and sometimes not so subtly) wrong. They are now slowly waking up to the reality that they can't do it that way if the distribution has to be truly uniform.

insane coder said...

> It looks like there would be more than enough material for a real article, if you can find the time for it.

Unfortunately I have a backlog of 3 articles that are partially completed, and I keep getting requests from others (many regarding LibreSSL, especially lately regarding fork() protection), and I still have real code that needs to be written.

> A whole generation of programmers has grown up with the modulo trick, without realizing that it's subtly (and sometimes not so subtly) wrong.

Indeed, however it's so much worse, and there's many libraries and books which claim to offer something better, and sometimes they are a bit better, but fall short of the ideal.

Of the ones I reviewed, the algorithms are usually far off the mark. The only reason why I corrected the one from SPC is because it was so close to correctness. Perhaps in later printings of the book they even corrected it, I know they used to publish a lot of errata.

As I mentioned above, there's also so many other issues with random related functions. Needing random strings is common, and some of the solutions I've seen are horribly broken. If you think uniformity is bad, you should see how some developers try to generate a password which matches minimum requirements of certain classes of characters (I saw one implementation which affixes A0! to the end of the string, and fills the rest with mersenne twister % 26 + 'a'). I've only ever seen one correct implementation for this. Nobody gets floating point random correct either in a portable manner, and I haven't come across a single ideal implementation yet.

insane coder said...

Jorden, regarding your comment about people not necessarily knowing why modulo is wrong, I did write about it here with pertinent details and an example, along with a number of other issues: http://insanecoding.blogspot.com/2014/05/dealing-with-randomness.html

I tried to be brief with this and the other issues, leaving just enough to get the point across. If there needs to be more elaboration on certain issues or solutions, I can do so.

Anonymous said...

Yes, you mentioned but the problem, but not really the solution, apart from "So in the above scenario, if 30 or 31 occurs, a new random number should be chosen". While that is technically correct, I would not trust most programmers to write a correct implementation based on that sentence alone. ;)

I came up with this approach independently (although, in my case, it was an optimization of my earlier "just keep requesting random numbers until the result is in range" approach, which (for single bytes) isn't actually as unreasonable a solution as it may sound) and wrote my own implementation a bit more than two years ago, only to find out that I wasn't the first to think of that solution. I know of several implementations, but you seem to have done more in-depth research.

It seems that the results of your testing would be an interesting basis for an article. Of course, I understand that you may have other priorities.

Alan Haugen said...
This comment has been removed by the author.
pragyachitra said...

Nice tips. Very innovative... Your post shows all your effort and great experience towards your work Your Information is Great if mastered very well.
angularjs-Training in velachery

angularjs Training in bangalore

angularjs Training in bangalore

angularjs Training in btm

angularjs Training in electronic-city

sai said...

I found this informative and interesting blog so i think so its very useful and knowledge able.I would like to thank you for the efforts you have made in writing this article.
Python training in marathahalli | Python training institute in pune

Unknown said...

myTectra a global learning solutions company helps transform people and organization to gain real, lasting benefits.Join Today.Ready to Unlock your Learning Potential !Read More

Unknown said...

Well researched article and I appreciate this. The blog is subscribed and will see new topics soon.
Java training in Chennai | Java training in USA |

Java training in Indira nagar | Java training in Rajaji nagar

priya rajesh said...

Learned a lot from your blog. Good creation and hats off to the creativity of your mind. Share more like this.
AWS Training in Chennai
AWS course in Chennai
AWS Certification in Chennai
DevOps Training in Chennai
RPA Training in Chennai
Angularjs Training in Chennai

Anonymous said...

Hey, Wow all the posts are very informative for the people who visit this site. Good work! We also have a Website. Please feel free to visit our site. Thank you for sharing.android developer future career

pavithra dass said...

Nice article. I liked very much. All the informations given by you are really helpful for my research. keep on posting your views.
Android Development Course in Chennai
Android app Development Course in Chennai
Android Training Institute in Chennai
Amazon Web Services Training in Chennai
AWS Training centers in Chennai
Best AWS Training in Chennai

REKHA PUNIA said...


If you live in Delhi and looking for a good and reliable vashikaran specialist in Delhi to solve all your life problems, then you are at right place. 
love marriage specialist in delhi

vashikaran specialist in delhi

love vashikaran specialist molvi ji

get love back by vashikaran

black magic specialist in Delhi

husband wife problem solution

mansur said...

Meet the best angularjs 7 training in Bangalore, trained 5000+ people, Get personalized training from industrial experts, with Hans on experience.

click here
click here
click here

mansur said...

meet the best angularjs 7 training in Bangalore, trained 5000+ people, Get personalized training from industrial experts, with Hans on experience.

Best angularjs 7 training in Bangalore

Unknown said...

Great!it is really nice blog information.after a long time i have grow through such kind of ideas.thanks for share your thoughts with us.
Cloud computing Training in Bangalore
Cloud computing courses in Anna Nagar
Cloud Computing Certification Training in T nagar
Cloud Computing Training in Sholinganallur

mansur said...

Thanks for your article, it was so informative as usual, keep it up!!
Meet the best training institute in Bangalore, trained 5000+ people,
Get personalized training from industrial experts with Hands on experience.

Top Angular JS training in Bangalore
Top React JS training in Bangalore
Top Node JS training in Bangalore

minakshi said...

Are you trying to move in or out of Jind? or near rohtak Find the most famous, reputed and the very best of all Packers and Movers by simply calling or talking to Airavat Movers and Packers

Packers And Movers in Jind

Packers And Movers in Rohtak

Movers And Packers in Rohtak


rama said...

I think this is the best article today about the future technology. Thanks for taking your own time to discuss this topic, I feel happy about that curiosity has increased to learn more about this topic.Artificial Intelligence Training in Bangalore. Keep sharing your information regularly for my future reference.

sureshbabus said...

Thanks for the awesome information.
Angularjs Training in Bangalore , Angular 2 Training in bangalore , Python Training in Bangalore

MyTraining said...

Great work,

for more details click here Best Training Institute in Bangalore

mobile application development said...

Thanks for the awesome article really great and fantastic blog, It was very helpful for me keep update.
enterprise mobile app development company
Mobile App Development Company in chennai
Mobile application development company in chennai

Jhonathan said...

Amazing Post. Your writing is very inspiring. Thanks for Posting.
Java application development company
Java development company
Java outsourcing company
Hire java developer
java web development services

digitalsourabh said...

C C
++ Classes in Bhopal

Nodejs Training in Bhopal
Big Data Hadoop Training in Bhopal

Digitalmarketing said...

I have perused your blog its appealing and noteworthy. I like it your blog.
digital marketing company in chennai,
digital marketing agency in india,
digital marketing company in chennai,
online marketing company in chennai,
digital marketing company in india,
digital marketing services,
digital marketing company,

MyTraining said...


Outstanding Information

AWS Training in Bangalore

Best AWS Training Institute in Bangalore

Software development company said...

I have scrutinized your blog its engaging and imperative. I like it your blog
custom application development services
Software development company
software application development company
offshore software development company
custom software development company

Software development company said...

I have scrutinized your blog its engaging and imperative. I like it your blog
custom application development services
Software development company
software application development company
offshore software development company
custom software development company

best ppc company in india said...

I have inspected your blog its associating with and essential. I like it your blog.
ppc services in india
best ppc company in india
ppc services india
ppc advertising services
ppc services company

sai said...

Really great post, I simply unearthed your site and needed to say that I have truly appreciated perusing your blog entries. I want to say thanks for great sharing.
Microsoft Azure online training
Selenium online training
Java online training
Python online training
uipath online training

Realtime Experts said...


waah Great Blog ! Really Appropriate


Would like to be an IT professionals..? and wants to join in Top MNCs..? then DO This Course

Devops Training in Bangalore

Anonymous said...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

https://www.kitsonlinetrainings.com/

Anonymous said...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

Ethical Hacking Interview Questions and Answers


ETL Testing Interview Questions and Answers


Exchange Server Interview Questions and Answers


Hadoop Interview Questions and Answers

Online.shoping.shop said...

https:www.mathstricksinhindi.in

Betty Hutt said...

Thanks for this blog, I really enjoyed reading your post.
avast contact number
McAfee Customer Service Number
norton contact number
yahoo mail phone number support
mozilla firefox support phone number

Riya Raj said...

Wonderful blog!!! More Useful to us... Thanks for sharing with us...
Selenium Training in Bangalore
Selenium Training in Coimbatore
Best Selenium Training Institute in Bangalore
best selenium training in coimbatore
RPA training in bangalore
Selenium Training in Bangalore
Oracle Training in Coimbatore
PHP Training in Coimbatore

gokul said...

Thank you so much for sharing this informative blog
data science interview questions pdf
data science interview questions online
data science job interview questions and answers
data science interview questions and answers pdf online
frequently asked datascience interview questions
top 50 interview questions for data science
data science interview questions for freshers
data science interview questions
data science interview questions for beginners
data science interview questions and answers pdf

Dale Morris said...

Microsoft Edge Customer Support
Mozilla firefox Customer Service Number
Sbcglobal Customer Service Phone Number
Yahoo Mail Customer Service Canada
Outlook Customer Service Phone Number

Unichrone said...

You may also see
PMP Certification Advantages

Unichrone said...

Nice information.
You may also try
ISO 27001 Lead Implementer Training Cairo Egypt

ISO 27001 Lead Auditor Training Cairo Egypt

ISO 27001 Lead Implementer Training Riyadh Saudi Arabia

ISO 27001 Lead Auditor Training Riyadh Saudi Arabia

Swetha K said...

The blog is very informative and the contents are very unique in here. I want more different post from your surpassing blog...
Tableau Training in Chennai
Tableau Course in Chennai
Power BI Training in Chennai
Job Openings in Chennai
Pega Training in Chennai
Linux Training in Chennai
Placement in Chennai
Soft Skills Training in Chennai
Appium Training in Chennai
Tableau Training in Velachery
Tableau Training in OMR

npn training said...

Big Data masters program is curated by the Hadoop industry. It is a masters program for those seeking to study Big Data. Hadoop is open source software that is used to store and protect Big Data.
https://www.npntraining.com/masters-program/big-data-architect-training/

Anbarasan14 said...

Thanks for sharing a worthy information. This is really helpful. Keep doing more.
Air Hostess Academy in Chennai
Air Hostess Training Institute in Chennai
Best Air Hostess Training Institute in Chennai
Air Hostess Course in Mumbai
Cabin Crew Institute in Mumbai
Cabin Crew Training Institute in Mumbai
Air Hostess Training in Bangalore
Cabin Crew Courses in Bangalore
Cabin Crew Training in Bangalore

Prwatech said...

I Got Job in my dream company with decent 12 Lacks Per Annum salary, I have learned this world most demanding course out there in the current IT Market from the data science training in Pune Providers who helped me a lot to achieve my dreams comes true. Really worth trying

Packers and Movers Varanasi said...

Jaibajrang Packers And Movers in Varanasi is an exclusive network of highly professional, reputed, best, leading, most experienced, genuine Packers and Movers services provider company in Varanasi. We provide packers and movers services in Varanasi, Household Services In Varanasi, Transporting Services In Varanasi, Car Bike Transport Services In Varanasi, Packers And Movers, Movers And Packers varanasi.

Jai Bajrang Transport Packers And Movers In Varanasi

Jai Bajrang Transport Household Services In Varanasi

Jai Bajrang Transporting Services In Varanasi

Jai Bajrang Transport Car, Bike Transport Services In Varanasi

Packers And Movers In Varanasi

Anonymous said...

For AWS training in Bangalore, Visit:
AWS training in Bangalore

Sakshi said...

Thanku.. so much for the blog..nice information..
For any kind of courses visit at SourceKode Training Institute

Mithun said...

Cool blog..
SAP Training in Chennai
SAP ABAP Training in Chennai
SAP Basis Training in Chennai
SAP FICO Training in Chennai
SAP MM Training in Chennai
SAP PM Training in Chennai
SAP PP Training in Chennai
SAP SD Training in Chennai

Digital_seo said...

Thank you for sharing useful information. Keep sharing more post
Selenium Training in Bangalore |
Software Testing Training in Bangalore|
Selenium Training in Marathahalli
Automation Testing Training in Bangalore |
Java Selenium Automation Training in Bangalore



Prwatech said...

The post is top quality ,it is very easy to understand to all readers.
tableau training institute in bangalore

Training for IT and Software Courses said...

It is very good and useful for students and developer .Learned a lot of new things from your post!Good creation ,thanks for give a good information at AWS.aws training in bangalore

Ascent World said...

Hi there,

Nice Article I really enjoyed this post Thanks For Sharing, check out this
Ascent WORLD is one of the World’s leading ‘total solutions’ providers, offering a simple, cost-effective route to ISO Certification in India & Sri Lanka. Ascent offers a wide range of certifications, like ISO 9001, ISO 14001, ISO 45001, HACCP, CE Mark and more.

With Ascent world, ISO certification is no longer a painful process
ISO Certification

Jack sparrow said...

Outstanding blog thanks for sharing such wonderful blog with us ,after long time came across such knowlegeble blog. keep sharing such informative blog with us. I Want share aboutmicrosoft exchange training videos.

svrtechnologies said...

Thanks for Posting such an useful info...keep blogging...

aws training videos

raju said...

nice post...
dominican republic web hosting
iran hosting
palestinian territory web hosting
panama web hosting
syria hosting
services hosting
afghanistan shared web hosting
andorra web hosting
belarus web hosting
brunei darussalam hosting

Kanagamani Hospital said...


This is so great. Thanks for sharing....

Durai Moorthy said...

This is a great article thanks for sharing this informative information.

aws Training in Bangalore
python Training in Bangalore
hadoop Training in Bangalore
angular js Training in Bangalore
bigdata analytics Training in Bangalore
python Training in Bangalore
aws Training in Bangalore

Johan said...

This is a really an awesome article which will be informative to everyone.

rpa training institutes in marathahalli

rpa course content

rpa training centres in bangalore

rpa computing course syllabus

rpa training

rpa training in marathahalli

My Class Training Bangalore said...

Really i appreciate the effort you made to share the knowledge. The topic here i found was really effective...

Best SAP HR Training in Bangalore
Best SAP BASIS Training in Bangalore
Best SAP HCM Training in Bangalore
Best SAP S4 HANA Simple Finance Training in Bangalore
Best SAP S4 HANA Simple Logistics Training in Bangalore

Henny Pavita said...
This comment has been removed by the author.
anuarun said...

ree internship in bangalore for computer science students
internship for aeronautical engineering
internship for eee students in hyderabad
internship in pune for computer engineering students 2018
kaashiv infotech internship fees
industrial training certificate format for mechanical engineering students
internship report on machine learning with python
internship for biomedical engineering students in chennai
internships in bangalore for cse
internship in coimbatore for ece

Vashikaran Sepecialist said...

Best Vashikaran Specialist in India, Vashikaran Specialist In India, offering effective Vashikaran solutions and services for Love, relationship, marriage, family, and husband-wife, who provide 100% vashikaran result in all vashikaran services Call to specialist 9899343854

Vashikaran Specialist In India
Vashikaran Specialist In India
Vashikaran Specialist In India
Vashikaran Specialist In India

shri said...

very nice...
internship report on python
free internship in chennai for ece students
free internship for bca
internship for computer science engineering students in india
internships in hyderabad for cse students 2018
electrical companies in hyderabad for internship
internships in chennai for cse students 2019
internships for ece students
inplant training in tcs chennai
internship at chennai

aj said...
This comment has been removed by the author.
aj said...

Snapdeal Winner List 2020 here came up with an Offer where you can win Snapdeal prize list by just playing a game & win prizes.
Snapdeal winner name also check the Snapdeal lucky draw

GlobalEmployees said...

Read my post here
Java programmer
Java programmer

MotoRidervlog said...

Hi sir, Thanks for sharing the information about business
TVS ntorq

GlobalEmployees said...

Find my blog post here
http://ttlink.com/bookmark/d3913482-056d-4e38-82e1-baa1036528ba
http://ttlink.com/bookmark/5931d44a-7910-41ec-9bab-f2b3082de030
http://ttlink.com/bookmark/3f596fd1-f173-4b25-ba95-0730a97ab3a8
Best Web Development Tools for Web Developers.

shakunthala said...

Thanks for sharing this awesome blog with us.
keep sharing!
aws training in Bangalore
linux training institutes in bangalore

MyTraining said...

Thank you for sharing. The data that you provided in the blog is informative and effective...
PHP Training in Bangalore - BTM | PHP Training Institutes | PHP Course Syllabus | PHP Course Fee - My Training Bangalore
- PHP Training in Bangalore, BTM - My Training Bangalore offers Best PHP Training with Expert Real-time PHP Trainers who have 8+ Years of Expertise,
Know more about PHP Details, Syllabus, Duration and Fees, we also provide placement assistance on successful completion of the Course.

Lili Jonath said...
This comment has been removed by the author.
Unknown said...


Thank you for sharing your post. It is awesome.

Arnsim
Best product and digital agency
Best web design and development company
Best digital marketing agency
Best customer support services
Best mobile app development company
Best design and development
Best design and development service provider
Arnsim - A web designand development company

Rajesh Anbu said...

Thanks for sharing this pretty post, it was good and helpful. Share more like this.
aws Training in Bangalore
python Training in Bangalore
hadoop Training in Bangalore
angular js Training in Bangalore
bigdata analytics Training in Bangalore
python Training in Bangalore
aws Training in Bangalore

Rajesh Anbu said...

I got useful information by reading this blogs
aws Training in Bangalore
python Training in Bangalore
hadoop Training in Bangalore
angular js Training in Bangalore
bigdata analytics Training in Bangalore
python Training in Bangalore
aws Training in Bangalore

Unknown said...

Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well as giving.

Spanish Language Training in Bangalore

Spanish Training Institute in Bangalore

Spanish Institute in Bangalore

Spanish Language Institute

Indian Institute of Spanish & Spanish Language

Spanish Learning

Spanish Language Institute in Bangalore

Spanish Courses

Spanish language course in Bangalore

learn Spanish language in Bangalore

Spanish language basics learning in Bangalore

learn Spanish grammar in Bangalore

Spanish learning centres contact addresses in Bangalore

Spanish grammer tuition phone numbers in Bangalore

reviews of Spanish language learning centres in Bangalore

ratings of Spanish spoken classes in Bangalore

svrtechnologies said...

Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content software testing course
to my vision, keep sharing.

Unknown said...

Poker online situs terbaik yang kini dapat dimainkan seperti Bandar Poker yang menyediakan beberapa situs lainnya seperti http://62.171.128.49/hondaqq/ , kemudian http://62.171.128.49/gesitqq/, http://62.171.128.49/gelangqq/, dan http://62.171.128.49/seniqq. yang paling akhir yaitu http://62.171.128.49/pokerwalet/. Jangan lupa mendaftar di panenqq silakan dicoba ya boss

svrtechnologies said...

I am inspired with your post writing style & how continuously you describe this topic. sap bi training After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.

Jack sparrow said...

Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision. i also want to share some infor mation regarding sap pp training and sap sd course.keep sharing.

Jack sparrow said...
This comment has been removed by the author.
Anand Shankar said...

Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained!


digital marketing blog
digital marketing bloggers
digital marketing blogs
digital marketing blogs in india
digital marketing blog 2020
digital marketing blog sites
skartec's digital marketing blog
skartec's blog
digital marketing course
digital marketing course in chennai
digital marketing training
skartec digital marketing academy

manimaran said...

thanks for your valuable information it is helpful for me, i like to share you something
training in velechery
Informatica IICS training in Velachery
Talend training in Velachery
Power Bi training in Velachery
SAP BO training in Velachery
Qlikview training in Velachery
Tableau training in Velachery
SSAS training in Velachery
SSRS training in Velachery
Cognos training in Velachery
sas training in Velachery

manimaran said...

thanks for your valuable information it is helpful for me, i like to share you something
training in medavakkam
SAP MM training institute in Medavakkam
SAP SD training institute in Medavakkam
Dot Net training institute in Medavakkam
PHP training institute in Medavakkam
Selenium training institute in Medavakkam
Big Data training institute in Medavakkam
SAP Abap training institute in Medavakkam
Salesforce Admin training institute in Medavakkam
CCNP training institute in Medavakkam

manimaran said...

online job support from india

Manual Testing Project Support from India
Workday Project Support from India
Salesforce Project Support from India
Android Project Support from India
DevOps Project Support from India
Java Project Support from India
ReactJS Project support from India
Angular Project Support from India
RPA Project Support from India
Blueprism Project Support from India

manimaran said...

T Shirt manufacturer in Tirupur
t shirt manufacturer in tirupur india
garments manufacturer in tirupur india
T Shirt Manufacturing in tirupur india

kids wear manufacturer in tirupur india
corporate t shirt manufacturer in tirupur india
school t shirt manufacturer in tirupur india
sports t shirt manufacturer in tirupur india
event t shirt manufacturer in tirupur india
uniform t shirt manufacturer in tirupur india
college t shirt manufacturer in tirupur india

Anurag Srivastava said...

Thank you so much for sharing such a useful information. I will definitely share this with others....Helpful for website i like the article...Indian RRB Recruitment 2020 Board has its own perks and advantages which is why it is preferred by young people. If you get a job in Indian railway recruitment, you can expect career growth and promotions...

svrtechnologies said...

I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.

sap fico training
sap fico videos




IICT Technologies said...

Cool
SAP Training in Chennai
Java Training in Chennai
CCNA Training in Chennai
Pearson Vue Exam Center in Chennai
QTP Training in Chennai
Selenium Training in Chennai
Hardware and Networking Training in Chennai
SAP ABAP Training in Chennai
SAP FICO Training in Chennai
AWS Training in Chennai

svrtechnologies said...

I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.

sap workflow online training

svrtechnologies said...

I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.

sap bw training

RIA Institute of Technology said...

This is a great article thanks for sharing this informative information...
Advanced Excel Training in Bangalore

vashikaran mantra said...


Husband Wife Problem Solution
Vashikaran Specialist In India
Get Lost Love Back By Vashikaran mantra
Vashikaran Mantra For Husband
Vashikaran Specialist In Melbourne
Vashikaran Specialist In Uk
Vashikaran Specialist In Canada

Jackson said...

this is outsanding ..i really loved every piece of information ..Thoroughly enjoyed reading it...i was searching for this information with lots of queries and this blog really helped me..

I also share some content on the technology.

For any support on Devops Training in Chennai

Thanks
Jackson

Oracle DBA Training in Chennai
Oracle DBA Course in Chennai
Oracle Training in Chennai
Oracle Certification Course in Chennai
Oracle DBA Training in Velachery
Microsoft Azure Training in chennai
azure solution architect training in chennai
azure certification course in chennai
azure training in chennai anna nagar
devops certification training in chennai
devops training in chennai velachery
devops training in chennai t nagar
Azure Training in chennai BITA Academy
devops training in chennai BITA Academy

svrtechnologies said...

This is so elegant and logical and clearly explained. Brilliantly goes through what could be a complex process and makes it obvious on servicenow training videos .

IICT said...

Informative blog post. Thanks for this wonderful Post.
SAP Training in Chennai
AWS Training in Chennai
Hardware and Networking Training in Chennai
QTP Training in Chennai
CCNA Training in Chennai

IICT said...

Informative blog post. Thanks for this wonderful Post.
SAP Training in Chennai
AWS Training in Chennai
Hardware and Networking Training in Chennai
QTP Training in Chennai
CCNA Training in Chennai

MotoRidervlog said...

Thanks for sharing the information and really explain everything

SSK Law Firm said...

Enjoyed reading the article above, really explains everything in detail, the article is very interesting and effective. Thank you and good luck…



Cheque Bounce Lawyers in Chennai
Civil Lawyers Lawyers in Chennai
Debt Recovery Lawyers in Chennai
Immigration Lawyers in Chennai
Divorce Lawyers in Chennai
Best Divorce Lawyers in Chennai
Dowry Harassement Lawyers in Chennai
Domestic Violence Alimony Lawyers in Chennai
Property Registration Lawyers in Chennai

Isa Bonita said...

A backlink is a link created when one website links to another. Backlinks are important to SEO & impact for higher ranking. In my 7+ years seo Career i see, without backlinks a website doesn't rank higher on google SERP.

Get Your 300+ High Quality DoFollow Backlinks Here!

Order Now with Full Confidence & 100% satisfaction.

CCC Service said...

Enjoyed reading the article above, really explains everything in detail, the article is very interesting and effective. Thank you and good luck…
'CCC Service
AC Service in Chennai
Fridge Service in Chennai
Washing Machine Service in Chennai
LED LCD TV Service in Chennai
Microwave Oven Service in Chennai'

rohan said...

Excellent Blog..Thanks for sharing..

uipath training in chennai , uipath classes near me , uipath certification training in chennai , uipath training in chennai BITA Academy , aws training in chennai
aws training in chennai BITA Academy , aws certification training in chennai , aws training center in chennai , aws training in chennai tambaram , aws training in chennai adayar , aws training in chennai vadapalani , aws course fees in chennai , aws course syllabus , aws training in chennai velachery

Unknown said...

Thanks for sharing amazing information.Gain the knowledge and hands-on experience.
Oneyes Technologies
Inplant Training in Chennai
Inplant Training in Chennai for CSE IT MCA
Inplant Training in Chennai ECE EEE EIE

Roku Com Link said...

You are sharing such an awesome and fantastic post with us. I appreciate your effort to share the knowledge. I am so happy to find this Blog which has informative and quality content for reading. Thanks for sharing. If it doesn’t work then, you have to read about the nitty gritty about Roku.com/link activation code from the Internet. Hopefully, then, you can connect the device with the Roku activation code accordingly.

Anonymous said...

Thank you for sharing this very useful
Aws Training In Bangalore

Civil Service Aspirants said...

Really i found this article more informative, thanks for sharing this article! Also Check here
Civil Service Aspirants
TNPSC Tutorial in English
TNPSC Tutorial in Tamil
TNPSC Notes in English
TNPSC Materials in English
tnpsc group 1 study materials

jeffrey cage said...



INSTEAD OF GETTING A LOAN,, I GOT SOMETHING NEW
Get $10,050 USD every week, for six months!

See how it works
Do you know you can hack into any ATM machine with a hacked ATM card??
Make up you mind before applying, straight deal...
Order for a blank ATM card now and get millions within a week!: contact us
via email address:: besthackersworld58@gmail.com or whats-app +1(323)-723-2568

We have specially programmed ATM cards that can be use to hack ATM
machines, the ATM cards can be used to withdraw at the ATM or swipe, at
stores and POS. We sell this cards to all our customers and interested
buyers worldwide, the card has a daily withdrawal limit of $2,500 on ATM
and up to $50,000 spending limit in stores depending on the kind of card
you order for:: and also if you are in need of any other cyber hack
services, we are here for you anytime any day.
Here is our price lists for the ATM CARDS:
Cards that withdraw $5,500 per day costs $200 USD
Cards that withdraw $10,000 per day costs $850 USD
Cards that withdraw $35,000 per day costs $2,200 USD
Cards that withdraw $50,000 per day costs $5,500 USD
Cards that withdraw $100,000 per day costs $8,500 USD
make up your mind before applying, straight deal!!!

The price include shipping fees and charges, order now: contact us via
email address::besthackersworld58@gmail.com or whats-app +1(323)-723-2568

Sylvia Bishop said...

Forex Signals, MT4 and MT5 Indicators, Strategies, Expert Advisors, Forex News, Technical Analysis and Trade Updates in the FOREX IN WORLD

Forex Signals Forex Strategies Forex Indicators Forex News Forex World

Tutorials said...

Thanks for the article. Its very useful. Keep sharing.   AWS training in chennai  |     AWS online course     AWS course online  

kartik swami said...

Acharya Ji has been providing genuine and effective black magic andvashikaran services for marriage, love, business, family, and other problems. He understands People’s Problem and gives effective solutions for Husband wife dispute, Get Lost Love back, Divorce solution, love problem solution, love marriage problem, Husband Wife problem, Inter cast love marriage and related fields. Acharya ji is also known as love marriage specialist, vashikaran specialist, and Vodoo spell caster expert. Acharaya ji is one of the best astrologers in India.

Devi said...

The post was really good.Thanks for sharing.oracle training in chennai

Indian Vashikaran Specialist said...

Pandit Ramakant Shastri is Indian Vashikaran specialist has expertise in Voodoo Black Magic Specialist, Get your Love Back, Love Guru Specialist, Get Back your Love, Love Breakup Solutions, Get Back your Love with Vashikaran, Get back your Lover, Women Love Vashikaran, Love with Vashikaran, Vashikaran Love Guru, Black Magic Guru Ji, Vashikaran Specialist, Vashikaran Baba Ji, Love Guru Tantrik, Black Magic Baba Ji, Kala Jadu Specialist, Women Vashikaran, Girlfriend Vashikaran, Boyfriend Vashikaran, Tantrik Baba Ji, Vashikaran Love Guru Ji, Astrology Solutions, Tantrik Expert Guru Ji, Black Magic Kala Jadu, Jyotish Specialist, Tantra Mantra Baba Ji, Vashikaran Specialist Guru Ji, Havan Pandit, Yagya Pooja Pandit, Kudli Dosh Solutions Expert, Match Making Specialist, Love Marriage Specialist, Indian Vashikaran Baba Ji, Horoscope Specialist, Kundli Dosh Solutions, Family Disputes Solutions, Love Sex Children Solutions, Vashikaran Expert, Higher Study Solutions, Tension Anxiety Solutions, Love Marriage Solutions, Business Loss Solutions, Foreign Visa Solutions, Illness Medical Problem Solutions, Kala Jadu Vashikaran, Property Disputes Solutions, Lottery Number Solutions, Get Your Love Back Mantra, Top Best Astrologers in Ludhiana, Amritsar, Jalandhar, Patiala, Bathinda, Ajitgarh, Hoshiarpur, Batala, Pathankot, Moga, Abohar, Khanna, Phagwwara, Muktsar, Barnala, Rajpura, Firozpur, Kapurthala, mandi gobindgarh, fatehgarh sahib, samrala, jagraon, mansa, sirhind, chandigarh, Mohali, panchkula, kharar, Punjab, Delhi, Haryana, Gurgaon, Utter Pradesh, Bihar, India, Canada, USA, Australia, UK, New Zealand, France, Norway, France, Nepal http://www.indianvashikaranspecialist.com contact numbers +91-9872458547, +91-9878958547 punditramakantshastri@gmail.com http://www.indianvashikaranspecialist.com

Shiv Shakti Jyotish Kendra said...

Pandit Deepak Kumar is Indian Vashikaran specialist in Get your Love Back, Love Breakup Solutions, Women Love Vashikaran, Black Magic Guru Ji, Love Guru Tantrik, Women Vashikaran, Girlfriend Vashikaran, Boyfriend Vashikaran, Tantrik Baba Ji, Astrology Solutions, Havan Pandit, Yagya Pooja Pandit, Kudli Dosh Solutions Expert, Match Making Specialist, Love Marriage Specialist, Business Loss Solutions, Foreign Visa Solutions, Illness Medical Problem Solutions, Kala Jadu Vashikaran, Property Disputes Solutions, Lottery Number Solutions in India, Punjab, Hoshiarpur, Canada, USA, Australia, UK, New Zealand, France, Nepal http://www.vashikaranhelpline.com contact numbers +91-9417683620, +91-9888821453 deepakrawal620@gmail.com http://www.vashikaranhelpline.com

laxmi said...

nice thanks for sharing...........................!
Exchange Server training
Google Cloud online training
Google Cloud training
IBM Intergration Bus online training
IBM Intergration Bus training
IBM Message Queue online training
IBM Message Queue training
Install shield online training
install shield training
Kubernetes online training
Kubernetes training
Linux Admin online training
Linux Admin training
Linux online training
Linux training
Load runner online training
Load runner training
MSBI online training

Anonymous said...

Thanks for provide great informatic and looking beautiful blog.
Python Interview Question And Answers
Web Api Interview Questions
OSPF Interview Questions And Answers

sugandha said...

Best vashikaran specialist for love problem solution. contact our free vashikaran expert for love now.

Brukkevin said...

I really appreciate the kind of topics you post here. Thanks for sharing us a great information that is actually helpful. Ben 10 Jacket

Mohd Sharique said...

Great blog found to be well written in a simple manner that everyone will understand. Big thumbs up for making such wonderful blog page!

Online training for big data
Big Data Hadoop Online Training

Ontario Security license said...

Shield Security Solutions Offers Security Guard License Training across the province of Ontario. Get Started Today!

Security Guard License | Security License | Ontario Security license | Security License Ontario | Ontario Security Guard License | Security Guard License Ontario

Technogeekscs said...

Fantastic blog! Thanks for sharing a very interesting post, I appreciate to blogger for an amazing post.

We are giving all Programming Courses such as You can

Register for a free Online Demo Classes

java courses in pune
Best Python Online Training
Online AWS Training
Online Data Science Training

Unknown said...

If you live in mumbai and looking for a good and reliable vashikaran specialist to solve all your life problems, https://vashikaran.biz/love-problem-solution-by-vashikaran-mantra/

krishan kumar said...

the website of Kstar Packers and Movers, your best choice for an experienced moving service provider in the city of Zirakpur and around Chandigarh.

https://www.kstarpackers.com/packers-and-movers-zirakpur/

Anuradha Astro Center said...

Black magic for control husband
Black magic for boyfriend

Cubestech said...

Thanks for the posts,

"best cloud service providers in chennai
Web development company in chennai
Best DevOps Service company in chennai
"

G Plus said...

I have been looking for this information for quite some times. Will look around your website .
Delhi Escort Service
Udaipur Escort Service
lucknow Escort Service
Noida Escort Service

Josh said...

aws training in chennai
Python training in Chennai
data science training in chennai
hadoop training in chennai

Daily Press Release said...

Pind daan in Gaya :- Gaya, also known as Bodh Gaya, is a city located in the Indian state of Bihar. This sacred city is located about 100 kilometers south of the state capital, Patna.

Daily Press Release said...
This comment has been removed by the author.
kagira said...

Thank you for sharing this great post !
Free piping course pdms training in Chennai

Brukkevin said...

It's actually a great and helpful piece of information. I am satisfied that you just shared this useful information for us. Womens Aviator Jacket

Anonymous said...

Enjoyed reading the article above, really explains everything in detail, the article is very interesting and effective. Thank you and good luck… 안전놀이터

Anonymous said...

really explains everything in detail, the article is very interesting and effective. Thank you and good luck 파워볼사이트

Anonymous said...

Really very happy to say, your post is very interesting to read. I never stop myself to say something about it. You’re doing a great job. Keep it up... 토토사이트

Anonymous said...

It's actually a great and helpful piece of information. I am satisfied that you just shared this useful information for us. 먹튀검증

Danny said...

Npower recruitment process is very easy. If you need to apply for Npower recruitment application portal in Nigeria then login to Npower portal first and register with the all the information you are asked to provide, then you are good to go.

black magic specialist in delhi said...

Get Love Problem Solution by Love Problem Specialist. 24x7 Available. Effective Solution Assured. Available on Call Consultation?. Call Now. Family Dispute Solution. Positive Black Magic. Positive Vashikaran.
Black Magic Specialist | Guru Maa Yashika
Black Magic Specialist | Guru Maa Yashika
lady Astrologer Guru Maa Yashika Devi ji
cast a love spell
Vashikaran Specialist Guru Maa Yashika

nayar said...

Wonderful blog.Thanks for sharing such a worthy information...
Digital Marketing Courses in Bangalore
Digital Marketing Course in Pune
Digital Marketing Course in Hyderabad
Digital Marketing Course in Gurgaon

SOFTWARE COMPANY said...

great post thanks for the sharing
web development agency
brisk logic is a software development agency in India Brisk Logic Covered: Ux/UI design, Web and Mobile development, Quality Assurance, Service Integration, Maintenance, AI/ML Solutions, IoT Solutions, IT Consulting, and other services to ensure a strong online presence for enterprises, boost sales, improve visibility and grow customer base

Malaivel Siddha Hospital said...

Wonderful post,This article have helped greatly continue writing...
Medicine for AIDS in India
HIV/AIDS Complete cure for siddha in India
HIV/AIDS Complete cure for siddha in Tamilnadu

self drilling anchor rock bolts india said...

Srons Engineers are Indian manufacturers exporters distributors of Rock Bolts, Self Drilling Anchor Bolt Systems, SDA Bolts, Threaded Rods, Hollow Bars, Fully Threaded Bars, Coil Rods, Hollow Rods, Tie Rods, Hollow Threaded Bars, Anchor Hollow Threaded Bars, Hot Rolled Threaded Rods, Hot Rolled Tie Rods, Forepoling Drilling System, Forepoling Drilling System, Tunnel Forepoling, Rock Drilling Systems, Pipe Roofing Solutions, Umbrella Pipe Roofing Systems, Tunnel Pipe Umbrella, High Tensile Thread Bars, MS Thread Rods, Plain Washers, Flat Washers, Spring Washers, Spring Lock Washers, Hex Nuts Fasteners, Hex Bolts Fasteners, Strut Support Systems, Spring Channel Nuts, Steel Fiber, Concrete Steel Fiber, hook end steel fiber, Loose steel fiber, Glued steel fiber, Tunnel Bolts, Minining Bolts, Tunnel Rock Bolts, Mining Rock Bolts, Coil Mining Fasteners, Tunnel Fasteners, Mining Fasteners, SAS 670 / 800 Bars, Din 125 Washer, Din 127 Spring Lock Washers etc in India Punjab Ludhiana. Our products are available in Material like Stainless Steel and Mild Steel and Finish of products like Hot Dip Galvanized & Zinc finish. We are exporting our products to UK, USA, Australia, Canada, UAE, Belgium, Denmark, Dubai, France, Germany, Ireland, Italy, Netherland, New Zealand, Norway, Saudi Arabia, Sri Lanka, Sweden, Bangladesh etc. +91-7508712122 www.sronsrockbolts.com
http://www.sronsrockbolts.com

self drilling anchor rock bolts india said...

Srons Engineers are Indian manufacturers exporters distributors of Rock Bolts, Self Drilling Anchor Bolt Systems, SDA Bolts, Threaded Rods, Hollow Bars, Fully Threaded Bars, Coil Rods, Hollow Rods, Tie Rods, Hollow Threaded Bars, Anchor Hollow Threaded Bars, Hot Rolled Threaded Rods, Hot Rolled Tie Rods, Forepoling Drilling System, Forepoling Drilling System, Tunnel Forepoling, Rock Drilling Systems, Pipe Roofing Solutions, Umbrella Pipe Roofing Systems, Tunnel Pipe Umbrella, High Tensile Thread Bars, MS Thread Rods, Plain Washers, Flat Washers, Spring Washers, Spring Lock Washers, Hex Nuts Fasteners, Hex Bolts Fasteners, Strut Support Systems, Spring Channel Nuts, Steel Fiber, Concrete Steel Fiber, hook end steel fiber, Loose steel fiber, Glued steel fiber, Tunnel Bolts, Minining Bolts, Tunnel Rock Bolts, Mining Rock Bolts, Coil Mining Fasteners, Tunnel Fasteners, Mining Fasteners, SAS 670 / 800 Bars, Din 125 Washer, Din 127 Spring Lock Washers etc in India Punjab Ludhiana. Our products are available in Material like Stainless Steel and Mild Steel and Finish of products like Hot Dip Galvanized & Zinc finish. We are exporting our products to UK, USA, Australia, Canada, UAE, Belgium, Denmark, Dubai, France, Germany, Ireland, Italy, Netherland, New Zealand, Norway, Saudi Arabia, Sri Lanka, Sweden, Bangladesh etc. +91-7508712122 www.sronsrockbolts.com

Hinditechnote said...

Nice article

android apkpure apk download

Dr.Shah Clinic said...


Male fertility doctor in chennai
Std clinic in chennai
Erectile dysfunction treatment in chennai
Premature ejaculation treatment in chennai
Small penis size treatment in chennai
Ivf clinic in chennai
Software IT Training Institute in Chennai

bruce wayne said...

Great blog.thanks for sharing such a useful information
Salesforce CRM Training in Chennai

Karthik said...

Nice Blog with a valid information. Thank you.
Cyber Security Course in Chennai
Cyber Security Training in Chennai

Awais Chughtai said...

what a informative and knowlegeable websites.Genius Professional Crack

Unknown said...

Thanks for sharing this helpful info with us. I'm glad that you shared this useful information with us. Please, keep us informed of this.
DgFlick Album Xpress Pro

Khan Honey said...

maxcrack

Unknown said...

I think Reflector Crack is a fantastic Post with great content. CRACKDUE is the best cracked software site for all Mac and Windows users throughout the world.

softskeygen said...


It’s great and fantastic piece. Keep it up as I look forward to read more from this website.
iSpoofer Crack Plus Activation Code

Unknown said...

O&O AutoBackup Professional Crack . is a great Post with useful information and great content. CRACK GURU is the most popular crack software website for Mac users and Windows users worldwide.

Unknown said...

Apeaksoft Data Recovery Crack is an excellent Post with Amazing content. All Mac and Windows users can download crack software from this KEYGENBRO site.

eddielydon said...

I trust you post again soon... leather peacoat

Easy Loan Mart said...

Hi....
While this decreases the quality of the randomness over time, it is still considered a cryptographically secure PRNG. What follows is the example of a function that can generate a 64 bit unsigned integer within a range, using /dev/urandom. Note that /dev/random and /dev/urandom are simply treated as files.
You are also read more Apply for Business Loan

Easy Loan Mart said...

Hi...
While this decreases the quality of the randomness over time, it is still considered a cryptographically secure PRNG. What follows is the example of a function that can generate a 64 bit unsigned integer within a range, using /dev/urandom. Note that /dev/random and /dev/urandom are simply treated as files.
You are also read more Best Housing Loan in India

Unknown said...

Hi, Thanks for sharing your blog.

Please review mine.

profile creation sites list | social bookmarking sites

Thanks

George Mark said...

Thanks for sharing such a informative blog with us thank you. Phoenix Valorant Jacket

Unknown said...

Payroll Software
payroll software singapore

Sandeep Prajapati said...

Accurate share market tips covering NSE, BSE and MCX. Stock market traders dealing in Indian share market can get Equity tips, future trading tips, nifty tips, options tips and commodity tips from sharetipsinfo for huge returns. Sharetipsinfo assures high returns on low investments.

Jobi Johnson said...

Put more informative things on this blog page, and thanks for sharing this. bella swan jacket

ShrutiMishra said...

Hi, nice article, Thanks for sharing this useful information, Learn SAP ABAP Certification Course in Affordable Fees from Experts with Job Placement Support.

SAP ABAP training in Indore
SAP ABAP Course in Bangalore

karlchrist impoludo said...






organic chemistry tutor
organic chemistry teacher

«Oldest ‹Older   1 – 200 of 211   Newer› Newest»